From arigo at codespeak.net Tue Nov 4 15:07:51 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 4 Nov 2003 15:07:51 +0100 (MET) Subject: [pypy-svn] rev 2160 - pypy/trunk/doc/devel Message-ID: <20031104140751.72D445B864@thoth.codespeak.net> Author: arigo Date: Tue Nov 4 15:07:50 2003 New Revision: 2160 Modified: pypy/trunk/doc/devel/howtosvn.txt Log: Moshez's instructions to install on Debian. Modified: pypy/trunk/doc/devel/howtosvn.txt ============================================================================== --- pypy/trunk/doc/devel/howtosvn.txt (original) +++ pypy/trunk/doc/devel/howtosvn.txt Tue Nov 4 15:07:50 2003 @@ -26,6 +26,8 @@ this will also need iconv_ MacOS X tar ball ++ Debian instructions below... + btw, HowToInstallServer_ sketches how to install a subversion server on Linux (not as easy as the client install). You don't need to install server side files to get your client going. Getting started @@ -40,7 +42,15 @@ download the tarball. unzip and untar it. Then type *./configure*. Then, as root, *make* followed by *make install*. Voil? ... a subversion client. -You can then go look at the files online_ with your browser, located at: http://codespeak.net/svn/pypy/trunk +A Debian package is available from backports_. Add the following line to ``/etc/apt/sources.list``:: + + deb http://fs.cs.fhm.edu/mirror/backports.org/debian stable subversion + +then install the package:: + + $ apt-get install subversion tools + +Note that you can always go look at the files online_ with your browser, located at: http://codespeak.net/svn/pypy/trunk But, you'll want to check out your own local copies to work on. Check out and Check in @@ -124,3 +134,4 @@ .. _coding-style: http://codespeak.net/pypy/doc/devel/coding-style.html .. _readme: http://codespeak.net/pypy/doc/readme.html .. _HowToInstallServer: http://codespeak.net/moin/pypy/moin.cgi/HowToInstallServer +.. _backports: http://www.backports.org From arigo at codespeak.net Tue Nov 4 15:11:31 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 4 Nov 2003 15:11:31 +0100 (MET) Subject: [pypy-svn] rev 2161 - pypy/trunk/doc/devel Message-ID: <20031104141131.73F035B864@thoth.codespeak.net> Author: arigo Date: Tue Nov 4 15:11:30 2003 New Revision: 2161 Modified: pypy/trunk/doc/devel/howtosvn.txt Log: Precisions Modified: pypy/trunk/doc/devel/howtosvn.txt ============================================================================== --- pypy/trunk/doc/devel/howtosvn.txt (original) +++ pypy/trunk/doc/devel/howtosvn.txt Tue Nov 4 15:11:30 2003 @@ -42,13 +42,13 @@ download the tarball. unzip and untar it. Then type *./configure*. Then, as root, *make* followed by *make install*. Voil? ... a subversion client. -A Debian package is available from backports_. Add the following line to ``/etc/apt/sources.list``:: +A Debian package is available from backports_. If you are using a *stable* Debian, first add the following line to ``/etc/apt/sources.list``:: deb http://fs.cs.fhm.edu/mirror/backports.org/debian stable subversion -then install the package:: +You can then install the package (same command for *testing* and *unstable*):: - $ apt-get install subversion tools + $ apt-get install subversion-tools Note that you can always go look at the files online_ with your browser, located at: http://codespeak.net/svn/pypy/trunk But, you'll want to check out your own local copies to work on. From arigo at codespeak.net Tue Nov 4 15:38:52 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 4 Nov 2003 15:38:52 +0100 (MET) Subject: [pypy-svn] rev 2162 - in pypy/trunk/doc: devel translation Message-ID: <20031104143852.A94955B86B@thoth.codespeak.net> Author: arigo Date: Tue Nov 4 15:38:51 2003 New Revision: 2162 Modified: pypy/trunk/doc/devel/howtosvn.txt pypy/trunk/doc/translation/controlflow.txt Log: Changed wording Modified: pypy/trunk/doc/devel/howtosvn.txt ============================================================================== --- pypy/trunk/doc/devel/howtosvn.txt (original) +++ pypy/trunk/doc/devel/howtosvn.txt Tue Nov 4 15:38:51 2003 @@ -42,13 +42,13 @@ download the tarball. unzip and untar it. Then type *./configure*. Then, as root, *make* followed by *make install*. Voil? ... a subversion client. -A Debian package is available from backports_. If you are using a *stable* Debian, first add the following line to ``/etc/apt/sources.list``:: +For Debian users:: - deb http://fs.cs.fhm.edu/mirror/backports.org/debian stable subversion + $ apt-get install subversion-tools -You can then install the package (same command for *testing* and *unstable*):: +People using Debian *stable* first need to add the following line to ``/etc/apt/sources.list`` (thanks backports_!):: - $ apt-get install subversion-tools + deb http://fs.cs.fhm.edu/mirror/backports.org/debian stable subversion Note that you can always go look at the files online_ with your browser, located at: http://codespeak.net/svn/pypy/trunk But, you'll want to check out your own local copies to work on. Modified: pypy/trunk/doc/translation/controlflow.txt ============================================================================== --- pypy/trunk/doc/translation/controlflow.txt (original) +++ pypy/trunk/doc/translation/controlflow.txt Tue Nov 4 15:38:51 2003 @@ -1,3 +1,58 @@ +FlowObjSpace +============ + +Introduction +------------ + +The FlowObjSpace generates a control-flow graph from a function. This graph also contains a trace of the individual operations, so that it is actually just an alternate representation for the function. + +The FlowObjSpace is an object space, which means that it exports the standard object space interface and it is driven by the interpreter. + +The basic idea is that if the interpreter is given a function, e.g.:: + + def f(n): + return 3*n+2 + +it will do whatever bytecode dispatching and stack-shuffling needed, during which it issues a sequence of calls to the object space. The FlowObjSpace merely records these calls (corresponding to "operations") in a structure called a basic block. To track which value goes where, the FlowObjSpace invents placeholder "wrapped objects" and give them to the interpreter, so that they appear in some next operation. + +For example, if the placeholder ``v1`` is given as the argument to the above function, the interpreter will call ``v2 = space.mul(space.wrap(3), v1)`` and then ``v3 = space.add(v2, space.wrap(2))`` and return ``v3`` as the result. During these calls the FlowObjSpace will record a basic block:: + + Block(v1): # input argument + v2 = mul(constant(3), v1) + v3 = add(v2, constant(2)) + + +Joining basic blocks +-------------------- + +A basic block ends in one of two cases: when the interpreters calls ``is_true()``, or when a joinpoint is reached. + +* A joinpoint is a specially marked position in the bytecode. This is the only bytecode dependency in FlowObjSpace. Intuitively, there should be one joinpoint at each bytecode position where two different paths can "join" together, e.g. the entry point of a loop. A joinpoint forces a basic block to end and the next one to begin. A snapshot of the current frame is taken (a FrameState) and recorded on the joinpoint. If the control flow later reaches this point again, we put a "backwards" jump to the old basic block that starts at this point. (The control flow is actually just a graph, with basic blocks pointing to each other, so there is not really a notion of "backwards".) + +* If the interpreter calls ``is_true()``, the FlowObjSpace doesn't generally know if the answer should be True or False, so it puts a conditional jump and generates two successor blocks for the current basic block. There is some trickery involved so that the interpreter is fooled into thinking that ``is_true()`` first returns False (and the subsequent operations are recorded in the first successor block), and later the *same* call to ``is_true()`` also returns True (and the subsequent operations go this time to the other successor block). + + +Passing variables between basic blocks +-------------------------------------- + +XXX + + +FlowExecutionContext +-------------------- + +The FlowExecutionContext is a modified ExecutionContext that drives the interpreter in a fashion that is quite different from the one needed for normal interpretation. XXX + + +Interface +--------- + +We make one instance of FlowExecutionContext per function to analyse. The instance has a cache of FrameStates to detect when the control is looping (``self.joinpoints``). XXX + + +Old stuff to remove +------------------- + :: Hello again From arigo at codespeak.net Tue Nov 4 19:35:36 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 4 Nov 2003 19:35:36 +0100 (MET) Subject: [pypy-svn] rev 2163 - in pypy/trunk/src/pypy: interpreter objspace/flow Message-ID: <20031104183536.799075B889@thoth.codespeak.net> Author: arigo Date: Tue Nov 4 19:35:25 2003 New Revision: 2163 Modified: pypy/trunk/src/pypy/interpreter/pyframe.py pypy/trunk/src/pypy/objspace/flow/framestate.py Log: Added __eq__ on FrameBlock class. Modified: pypy/trunk/src/pypy/interpreter/pyframe.py ============================================================================== --- pypy/trunk/src/pypy/interpreter/pyframe.py (original) +++ pypy/trunk/src/pypy/interpreter/pyframe.py Tue Nov 4 19:35:25 2003 @@ -105,6 +105,17 @@ self.handlerposition = handlerposition self.valuestackdepth = frame.valuestack.depth() + def __eq__(self, other): + return (self.__class__ is other.__class__ and + self.handlerposition == other.handlerposition and + self.valuestackdepth == other.valuestackdepth) + + def __ne__(self, other): + return not (self == other) + + def __hash__(self): + return hash((self.handlerposition, self.valuestackdepth)) + def cleanupstack(self, frame): for i in range(self.valuestackdepth, frame.valuestack.depth()): frame.valuestack.pop() Modified: pypy/trunk/src/pypy/objspace/flow/framestate.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/framestate.py (original) +++ pypy/trunk/src/pypy/objspace/flow/framestate.py Tue Nov 4 19:35:25 2003 @@ -51,8 +51,7 @@ # nonmergeable states assert isinstance(other, FrameState) assert len(self.mergeable) == len(other.mergeable) - # XXX assert self.nonmergeable == other.nonmergeable - # XXX this requires a proper __eq__ on the blockstack items + assert self.nonmergeable == other.nonmergeable for w1, w2 in zip(self.mergeable, other.mergeable): if not (w1 == w2 or (isinstance(w1, Variable) and isinstance(w2, Variable))): From arigo at codespeak.net Tue Nov 4 19:43:18 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 4 Nov 2003 19:43:18 +0100 (MET) Subject: [pypy-svn] rev 2164 - pypy/trunk/src/pypy/interpreter Message-ID: <20031104184318.93B8B5B88A@thoth.codespeak.net> Author: arigo Date: Tue Nov 4 19:43:16 2003 New Revision: 2164 Modified: pypy/trunk/src/pypy/interpreter/pyframe.py Log: Deleted a special-purpose exception class and replaced it with 'return'. Modified: pypy/trunk/src/pypy/interpreter/pyframe.py ============================================================================== --- pypy/trunk/src/pypy/interpreter/pyframe.py (original) +++ pypy/trunk/src/pypy/interpreter/pyframe.py Tue Nov 4 19:43:16 2003 @@ -127,6 +127,7 @@ def unroll(self, frame, unroller): "Clean up a frame when we abnormally exit the block." self.cleanupstack(frame) + return False # continue to unroll class LoopBlock(FrameBlock): @@ -140,12 +141,13 @@ frame.blockstack.push(self) jump_to = unroller.args[0] frame.next_instr = jump_to - raise StopUnrolling + return True # stop unrolling self.cleanupstack(frame) if isinstance(unroller, SBreakLoop): # jump to the end of the loop frame.next_instr = self.handlerposition - raise StopUnrolling + return True # stop unrolling + return False class ExceptBlock(FrameBlock): @@ -168,7 +170,8 @@ frame.valuestack.push(w_value) frame.valuestack.push(w_type) frame.next_instr = self.handlerposition # jump to the handler - raise StopUnrolling + return True # stop unrolling + return False def app_normalize_exception(etype, evalue): # XXX should really be defined as a method on OperationError, @@ -215,7 +218,7 @@ frame.valuestack.push(frame.space.w_None) frame.valuestack.push(frame.space.w_None) frame.next_instr = self.handlerposition # jump to the handler - raise StopUnrolling + return True # stop unrolling ### Internal exceptions that change the control flow ### @@ -240,13 +243,12 @@ """ def action(self, frame, last_instr, executioncontext): "Default unroller implementation." - try: - while not frame.blockstack.empty(): - block = frame.blockstack.pop() - block.unroll(frame, self) + while not frame.blockstack.empty(): + block = frame.blockstack.pop() + if block.unroll(frame, self): + break + else: self.emptystack(frame) - except StopUnrolling: - pass def emptystack(self, frame): "Default behavior when the block stack is exhausted." @@ -275,14 +277,9 @@ w_returnvalue = self.args[0] raise ExitFrame(w_returnvalue) -class StopUnrolling(Exception): - "Signals the end of the block stack unrolling." - class ExitFrame(Exception): """Signals the end of the frame execution. The argument is the returned or yielded value, already wrapped.""" class BytecodeCorruption(ValueError): """Detected bytecode corruption. Never caught; it's an error.""" - - From arigo at codespeak.net Tue Nov 4 20:03:38 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 4 Nov 2003 20:03:38 +0100 (MET) Subject: [pypy-svn] rev 2165 - pypy/trunk/src/pypy Message-ID: <20031104190338.0FAB45B88D@thoth.codespeak.net> Author: arigo Date: Tue Nov 4 20:03:37 2003 New Revision: 2165 Modified: pypy/trunk/src/pypy/TODO Log: Removed completed tasks, added a note about doc/translation/annotation.txt. Modified: pypy/trunk/src/pypy/TODO ============================================================================== --- pypy/trunk/src/pypy/TODO (original) +++ pypy/trunk/src/pypy/TODO Tue Nov 4 20:03:37 2003 @@ -1,9 +1,5 @@ -Done: refactor the flowmodel to translator/flowmodel.idea - (Armin and Holger plan on doing this next saturday) - -Task: announce the AmsterdamSprint (maybe together with the next task) -Task: announce the EU proposal - (Armin and Holger plan on doing this sunday) +Task: review and implement the annotation procedure described in + doc/translation/annotation.txt Task: update pypy homepage From arigo at codespeak.net Tue Nov 4 20:42:47 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 4 Nov 2003 20:42:47 +0100 (MET) Subject: [pypy-svn] rev 2166 - pypy/trunk/doc/translation Message-ID: <20031104194247.66EEC5B88F@thoth.codespeak.net> Author: arigo Date: Tue Nov 4 20:42:46 2003 New Revision: 2166 Modified: pypy/trunk/doc/translation/annotation.txt Log: Separated again the control flow generation from the annotation pass. Added Holger's suggestion about polymorphic code. Modified: pypy/trunk/doc/translation/annotation.txt ============================================================================== --- pypy/trunk/doc/translation/annotation.txt (original) +++ pypy/trunk/doc/translation/annotation.txt Tue Nov 4 20:42:46 2003 @@ -1,5 +1,10 @@ -Mixed control flow / annotation pass -==================================== +The annotation pass +=================== + +Let's assume that the control flow graph building pass can be +done entierely before the annotation pass. (See notes at the +end for why we'd like to mix them.) + Factorial --------- @@ -13,15 +18,30 @@ else: return 1 -Suppose that we know that ``f`` takes an ``int`` argument. -We start with the flow objspace applied to the entry point:: +The flow objspace gives us the following graph (after the +simplification that makes ``simple_call``):: StartBlock(v1): v2 = ge(v1, 2) exitswitch(v2) + link "True" to Block2(v1) + link "False" to Block3 -We suspend the flow objspace here, and we immediately do the -type inference on this block:: + Block2(v3): + v4 = sub(v3, 1) + v7 = simple_call(f, v4) + v8 = mul(v3, v7) + jump to ReturnBlock(v8) + + Block3: + v9 = 1 + jump to ReturnBlock(v9) + + ReturnBlock(retval): + (empty, just returns retval) + +Suppose that we know that ``f`` takes an ``int`` argument. +We start type inference on the first block:: Analyse(StartBlock): v1 ----> X1 type(X1)=int @@ -34,34 +54,8 @@ about the unknown heap objects. The arrows represent binding from variables to objects. -We perform the type inference early because it may give interesting -information about ``v2``, the variable whose truth-value determines in -which block we must continue. In this case we don't know if ``v2`` will -be True or False, but in some cases type inference can help (for -example, in code like ``a, b = c, d`` the type inference can tell that -the right-hand tuple is of length two and so no ValueError will be -thrown for unpacking a tuple of the wrong length). - -Let's come back to StartBlock. We add an exit corresponding to the case -``v2==True``, jumping to Block2 which we flow-analyse now:: - - Block2(v3): - v4 = sub(v3, 1) - v5 = newtuple(v4) - v6 = newdict() - v7 = call(f, v5, v6) - v8 = mul(v3, v7) - jump to ReturnBlock(v8) - -This is simplified into:: - - Block2(v3): - v4 = sub(v3, 1) - v7 = simple_call(f, v4) - v8 = mul(v3, v7) - jump to ReturnBlock(v8) - -Type inference:: +After StartBlock, we proceed to the type inference of its exits; +first Block2:: Analyse(Block2): v3 ------------> X1 # copied from StartBlock @@ -70,22 +64,14 @@ It fails at the simple_call to f, because we don't know yet anything about the return value of f. We suspend the analysis of Block2 and -resume at some other non-blocked point -- for example, we can now -consider adding an exit to StartBlock for the case ``v2==False``, -jumping to Block3:: - - Block3: - v9 = 1 - jump to ReturnBlock(v9) +resume at some other non-blocked point -- in this case, the other exit +from the StackBlock, which is jumping to Block3:: Analyse(Block3): v9 --------> 1 # and we have type(1)=int automatically Then we proceed to ReturnBlock:: - ReturnBlock(retval): - (empty, just returns retval) - Analyse(ReturnBlock): retval --------> 1 @@ -154,17 +140,17 @@ A program of more than one function is analysed in exactly the same way, starting from an entry point and following calls. We have a cache of all -the blocks that the flow objspace produced, and a list of pending blocks +the flowgraphs that the flow objspace produced, and a list of pending blocks that we have to type-analyse. When the analysis of a block temporarily fails (as above for the first recursive call to ``f``) we move the block -at the end of the pending list. There is only one heap of annotations -for the whole program, so that we can track where the objects come from -and go through the whole program. (This makes separate compilation very -difficult, I guess.) +back into the pending list. There is only one heap of annotations for the +whole program, so that we can track where the objects come from and go +through the whole program. (This makes separate compilation very difficult, +I guess.) -Empty lists ------------ +Empty lists and mutable objects +------------------------------- Nothing special is required for empty lists. Let's try:: @@ -184,10 +170,10 @@ v3 = simple_call(g, v1, 6) Analyse(F_StartBlock): - v1 -------> X1 type(X1)=list len(X1)=0 getitem(X1,?)=? + v1 -------> X1 type(X1)=list len(X1)=0 getitem(X1,*)=? v2 -------> crash -The ``?`` is a special value meaning ``no analysis information``. The type analysis fails because of the calls to ``g``, but it triggers the analysis of ``g`` with the input arguments' annotations:: +The ``?`` is a special value meaning ``no analysis information``, and ``*`` is a special catch-all value. The type analysis fails because of the calls to ``g``, but it triggers the analysis of ``g`` with the input arguments' annotations:: G_StartBlock(v4, v5): v6 = getattr(v4, 'append') @@ -220,7 +206,7 @@ And so this time the list ``X1`` is updated with:: - getitem(X1,?)=X5 + getitem(X1,*)=X5 and now we know that we have a list of integers. @@ -236,3 +222,33 @@ themselves use a representation that depends on all the lists that could come at this point. All these places and lists will use a common, most general representation. + + +Polymorphism and mixed flowing/inference +---------------------------------------- + +We might eventually mix type inference and control flow generation a bit +more than described above. The annotations could influence the generation +of the graph. + +The most interesting influence would be to occasionally prevent two +FrameStates from being merged. This would result in a bigger control flow +graph in which several basic blocks can contain the operations about the +same bytecode positions, with different annotations. In particular, a +quite interesting idea is to disallow two states to be merged if the +resulting intersection of annotations is too poor -- say if it would make +genpyrex.py use the fall-back generic object type, which is not available +to other genxxx.py. + +The result is that we will automatically generate several specialized +version of the RPython code when it is meant to be polymorphic. For +example, in a function such as:: + + def push(stack, item): + stack.append(item) + +the different entry points, specifying quite different type annotations +for ``item``, are all unmergeable, as merging them would result in +insufficently many annotations left. By contrast, in the factorial +example above, all merges are fine because they conserve at least the +``type(X)=int`` annotation. From sanxiyn at codespeak.net Wed Nov 5 18:10:33 2003 From: sanxiyn at codespeak.net (sanxiyn at codespeak.net) Date: Wed, 5 Nov 2003 18:10:33 +0100 (MET) Subject: [pypy-svn] rev 2181 - pypy/trunk/src/pypy Message-ID: <20031105171033.0F3535B8B6@thoth.codespeak.net> Author: sanxiyn Date: Wed Nov 5 18:10:32 2003 New Revision: 2181 Modified: pypy/trunk/src/pypy/TODO Log: TODO: import related Modified: pypy/trunk/src/pypy/TODO ============================================================================== --- pypy/trunk/src/pypy/TODO (original) +++ pypy/trunk/src/pypy/TODO Wed Nov 5 18:10:32 2003 @@ -1,5 +1,5 @@ Task: review and implement the annotation procedure described in - doc/translation/annotation.txt + doc/translation/annotation.txt Task: update pypy homepage @@ -8,11 +8,12 @@ Task: fix interpreter-level object introspection from app-level (make dis.dis(dis.dis) goal work as a side effect) -Task: provide an importer that can include packages (we can't at the - moment) +Task: provide an importer that can import packages + consider PEP 302, new import hooks + try to write as much as possible in app-level + how would PyPy import CPython extension when it runs on top of CPython? Task: slicing with assigments & more small things (finish StdObjSpace review!) - lst[::-2] = lst2 Task: try tests of CPython in PyPy From arigo at codespeak.net Tue Nov 11 16:17:21 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 11 Nov 2003 16:17:21 +0100 (MET) Subject: [pypy-svn] rev 2194 - in pypy/trunk/src/pypy/translator: . test Message-ID: <20031111151721.535A95BD4D@thoth.codespeak.net> Author: arigo Date: Tue Nov 11 16:17:19 2003 New Revision: 2194 Added: pypy/trunk/src/pypy/translator/annheap.py (contents, props changed) - copied, changed from rev 2191, pypy/trunk/src/pypy/translator/annset.py pypy/trunk/src/pypy/translator/test/test_annheap.py (contents, props changed) - copied, changed from rev 2191, pypy/trunk/src/pypy/translator/test/test_annset.py Log: First try at an AnnotationHeap for the implementation of doc/translator/annotation.txt. Copied: pypy/trunk/src/pypy/translator/annheap.py (from rev 2191, pypy/trunk/src/pypy/translator/annset.py) ============================================================================== --- pypy/trunk/src/pypy/translator/annset.py (original) +++ pypy/trunk/src/pypy/translator/annheap.py Tue Nov 11 16:17:19 2003 @@ -3,36 +3,33 @@ from pypy.objspace.flow.model import Variable, Constant, SpaceOperation -class Cell: - """A logical variable. A Cell is an initially empty place that can - later contain a Variable or a Constant.""" - - # a Cell is "empty" if self.content is None, - # a Cell is "ground" otherwise. Once a Cell is ground its content - # cannot be changed any more. - - # Multiple empty Cells can be "shared"; a group of shared Cells act - # essentially like a single Cell in that setting one Cell content - # will give all Cells in the group the same content. +class XCell: + """A placeholder for a heap object contained in an AnnotationHeap. + It represents an object that will actually appear at run-time in the heap. + XCells are the arguments and return value of SpaceOperations when + used as annotations.""" + + counter = 0 + + # Multiple XCells can be "shared"; a group of shared cells + # act essentially like a single cell (they become all equal). - def __init__(self): - self.content = None - self.shared = [] # list of weakrefs to Cells - # defining a group of shared cells + def __init__(self, name=None): + if not name: + name = 'X%d' % XCell.counter + XCell.counter += 1 + self.name = name + self.shared = [] # list of weakrefs to XCells + # defining a group of shared cells. def __repr__(self): - if self.content is None: - first = self.cellsingroup()[0] - return '' % (id(first),) - else: - return '' % (self.content,) + names = [cell.name for cell in self.cellsingroup()] + names.sort() + return '=='.join(names) def __eq__(self, other): - "Two sharing cells are identical." - if isinstance(other, Cell): - return self.is_shared(other) - else: - return self.content == other + "Two sharing cells are equal." + return isinstance(other, XCell) and self.is_shared(other) def __ne__(self, other): return not (self == other) @@ -50,26 +47,13 @@ self.shared = [weakref.ref(self)] return self.shared - def set(self, content): - if isinstance(content, Cell): - self.share(content) - elif self.content is None: - for c in self.cellsingroup(): - c.content = content - elif self.content != content: - raise ValueError, "cannot change the content of %r" % self - def is_shared(self, other): "Test if two cells are shared." return self.shared is other.shared def share(self, other): - "Make two Cells share content." + "Make two cells shared." if not self.is_shared(other): - if self.content is not None: - other.set(self.content) - elif other.content is not None: - self.set(other.content) lst1 = self.getsharelist() lst2 = other.getsharelist() for s in lst2: @@ -79,153 +63,131 @@ lst1.append(s) -class AnnotationSet: - - def __init__(self, annlist=[]): - self.byfunctor = {} - for ann in annlist: - self.add(ann) +class XConstant(XCell): + """A fully determined XCell. For immutable constants.""" - def copy(self): - a = AnnotationSet() - for functor, lst in self.byfunctor.items(): - a.byfunctor[functor] = lst[:] - return a + def __init__(self, value): + XCell.__init__(self) + self.value = value - def __repr__(self): - fulllist = list(self.enumerate()) - if fulllist: - lines = (['']) - else: - lines = ['' % (id(self),)] - return '\n'.join(lines) + def __eq__(self, other): + "Two constants with the same value are equal." + return (isinstance(other, XConstant) and self.value == other.value + or XCell.__eq__(self, other)) - def __len__(self): - result = 0 - for lst in self.byfunctor.values(): - result += len(lst) - return result - def match(self, pattern): - """Test if the annotation 'pattern' is present in the set. - This function sets all empty Cells of 'pattern', but it does not - change Cells in 'self'. All Cells of 'pattern' must be fresh.""" - functor = pattern.opname, len(pattern.args) - for ann in self.byfunctor.get(functor, []): - if same_functor_assign(pattern, ann): - return True - return False - - def intersect(self, otherset): - """Kill annotations in 'self' that are not present in 'otherset'. - It may set some Cells in 'self', but it does not change 'otherset'.""" - for annlist in self.byfunctor.values(): - for i in range(len(annlist)-1, -1, -1): - ann = annlist[i] - if not otherset.match(ann): - del annlist[i] - - def add(self, pattern): - """Add 'pattern' into 'self'. It may set some Cells in 'self' instead - of adding a new entry.""" - functor = pattern.opname, len(pattern.args) - annlist = self.byfunctor.setdefault(functor, []) - for ann in annlist: - if same_functor_assign(ann, pattern): - pattern = ann - break - else: - annlist.append(pattern) +# The more annotations about an XCell, the least general +# it is. Extreme case: *all* possible annotations stand for an +# object that cannot exist (e.g. the return value of a function +# that never returns or we didn't see return so far). +# This is specified by using nothingyet instead of a real XCell(). +# Conversely, *no* annotation stands for any object. - def enumerate(self, renaming=None): - """Yield a copy of all annotations in the set, possibly renaming - their variables according to a map {Variable: [list-of-Variables]}.""" - if renaming is None: - def renameall(list_w): - return [list_w] - else: - def rename(w): - if isinstance(w,Constant): - return [w] - else: - return renaming.get(w, []) - def renameall(list_w): - if list_w: - for w in rename(list_w[0]): - for tail_w in renameall(list_w[1:]): - yield [w] + tail_w - else: - yield [] - for lst in self.byfunctor.values(): - for ann in lst: - # we translate a single SpaceOperation(...) into either - # 0 or 1 or multiple ones, by replacing each variable - # used in the original operation by (in turn) any of - # the variables it can be renamed into - for list_w in renameall([ann.result] + ann.args): - result = list_w[0] - args = list_w[1:] - yield SpaceOperation(ann.opname,args,result) +nothingyet = XCell('nothingyet') - __iter__ = enumerate - ### convenience methods ### +class AnnotationHeap: + """An annotation heap is a (large) family of annotations. + An annotation is a SpaceOperation with XCells as arguments and result.""" + + def __init__(self, annlist=[]): + self.annlist = [] # List of annotations XXX optimize + for ann in annlist: + self.add(ann) - def set_type(self, v, type): - self.add(SpaceOperation('type', [v], Constant(type))) + def dump(self): # debugging + for ann in self.enumerate(): + print ann + + def add(self, annotation): + """Register an annotation into the heap.""" + if annotation not in self.annlist: + self.annlist.append(annotation) + + def enumerate(self): + """Enumerates all annotations in the heap.""" + return iter(self.annlist) - def get_type(self, v): - if isinstance(v, Constant): - return type(v.value) - c = Cell() - self.match(SpaceOperation('type', [v], c)) - if isinstance(c.content, Constant): - return c.content.value - else: - return None + __iter__ = enumerate - def get_opresult(self, opname, args): - c = Cell() - self.match(SpaceOperation(opname, args, c)) - if isinstance(c.content, Constant): - return c.content.value + def set_type(self, cell, type): + """Register an annotation describing the type of the object 'cell'.""" + self.add(SpaceOperation('type', [cell], XConstant(type))) + + def get_type(self, cell): + """Get the type of 'cell', as specified by the annotations, or None.""" + c = self.get_opresult('type', [cell]) + if isinstance(c, XConstant): + return c.value else: return None + def get_opresult(self, name, args): + """Return the Cell with the annotation 'name(args) = cell', + or None if there is no such annotation, or several different ones.""" + result = None + for ann in self.annlist: + if ann.opname == name and ann.args == args: + if result is None: + result = ann.result + elif ann.result != result: + return None + return result -def annotation_assign(ann1, ann2): - """Assignment (ann1 = ann2). All empty cells in 'ann1' are set to the - value found in 'ann2'. Returns False if the two annotations are not - compatible.""" - functor1 = ann1.opname, len(ann1.args) - functor2 = ann2.opname, len(ann2.args) - return functor1 == functor2 and same_functor_assign(ann1, ann2) - -def same_functor_assign(ann1, ann2): - """Assignment (ann1 = ann2). All empty cells in 'ann1' are set to the - value found in 'ann2'. Returns False if the variables and constants - in the two annotations are not compatible. Assumes that the two - annotations have the same functor.""" - pairs = zip(ann1.args + [ann1.result], ann2.args + [ann2.result]) - for a1, a2 in pairs: - v1 = deref(a1) - if not isinstance(v1, Cell): - v2 = deref(a2) - if not isinstance(v2, Cell) and v2 != v1: - return False - # match! Set the Cells of ann1... - for a1, a2 in pairs: - v1 = deref(a1) - if isinstance(v1, Cell): - v1.set(a2) - return True - -def deref(x): - """If x is a Cell, return the content of the Cell, - or the Cell itself if empty. For other x, return x.""" - if isinstance(x, Cell) and x.content is not None: - return x.content - else: - return x + def merge(self, oldcell, newcell): + """Update the heap to account for the merging of oldcell and newcell. + Return (resultcell, changeflag) where resultcell is the merged cell. + changeflag is false only if the merged cell is equal to oldcell and + no annotations about oldcell have been dropped.""" + if newcell is nothingyet or newcell == oldcell: + return oldcell, False + elif oldcell is nothingyet: + return newcell, True + else: + # find the annotations common to oldcell and newcell + common = [] + deleting = False # means deleting an annotation about oldcell + for ann in self.annlist: + if oldcell in ann.args or oldcell == ann.result: + test1 = rename(ann, oldcell, newcell) + test2 = rename(ann, newcell, oldcell) # may equal 'ann' + if test1 in self.annlist and test2 in self.annlist: + common.append(test1) + else: + deleting = True + # the involved objects are immutable if we have both + # 'immutable() -> oldcell' and 'immutable() -> newcell' + if SpaceOperation('immutable', [], newcell) in common: + # for immutable objects we can create a new cell if necessary + if not deleting: + return oldcell, False # nothing must be removed from oldcell + else: + resultcell = XCell() # invent a new cell + for ann in common: + self.add(rename(ann, newcell, resultcell)) + return resultcell, True + else: + # for mutable objects we must identify oldcell and newcell, + # and only keep the common annotations + newcell.share(oldcell) + # add to 'common' all annotations that don't talk about oldcell + # (nor newcell, but at this point oldcell == newcell) + for ann in self.annlist: + if not (oldcell in ann.args or oldcell == ann.result): + common.append(ann) + # 'common' is now the list of remaining annotations + self.annlist[:] = common + return oldcell, deleting + + +def rename(ann, oldcell, newcell): + "Make a copy of 'ann' in which 'oldcell' has been replaced by 'newcell'." + args = [] + for a in ann.args: + if a == oldcell: + a = newcell + args.append(a) + a = ann.result + if a == oldcell: + a = newcell + return SpaceOperation(ann.opname, args, a) Copied: pypy/trunk/src/pypy/translator/test/test_annheap.py (from rev 2191, pypy/trunk/src/pypy/translator/test/test_annset.py) ============================================================================== --- pypy/trunk/src/pypy/translator/test/test_annset.py (original) +++ pypy/trunk/src/pypy/translator/test/test_annheap.py Tue Nov 11 16:17:19 2003 @@ -2,67 +2,17 @@ import autopath from pypy.tool import test -from pypy.translator.annset import Cell, AnnotationSet -from pypy.objspace.flow.model import Variable, Constant, SpaceOperation +from pypy.translator.annheap import XCell, XConstant, AnnotationHeap, nothingyet +from pypy.objspace.flow.model import SpaceOperation -class TestCell(test.IntTestCase): - - def test_set(self): - c1 = Cell() - v1 = Variable('v1') - c1.set(v1) - self.assertEquals(c1.content, v1) - c2 = Cell() - k2 = Constant(123) - c2.set(k2) - self.assertEquals(c2.content, k2) - self.assertRaises(ValueError, c1.set, k2) - self.assertRaises(ValueError, c2.set, v1) - self.assertRaises(ValueError, c1.set, c2) - self.assertRaises(ValueError, c2.set, c1) - - def test_share1(self): - k1 = Constant(-123) - c1 = Cell() - c2 = Cell() - c1.share(c2) - c2.set(k1) - self.assertEquals(c1.content, k1) - self.assertEquals(c2.content, k1) - - def test_share2(self): - k1 = Constant(-123) - c1 = Cell() - c2 = Cell() - c2.set(k1) - c1.share(c2) - self.assertEquals(c1.content, k1) - self.assertEquals(c2.content, k1) - - def test_share3(self): - k1 = Constant(-123) - c1 = Cell() - c2 = Cell() - c1.share(c2) - c1.set(k1) - self.assertEquals(c1.content, k1) - self.assertEquals(c2.content, k1) - - def test_share3(self): - k1 = Constant(-123) - c1 = Cell() - c2 = Cell() - c1.set(k1) - c1.share(c2) - self.assertEquals(c1.content, k1) - self.assertEquals(c2.content, k1) +class TestXCell(test.IntTestCase): def test_is_shared(self): - c1 = Cell() - c2 = Cell() - c3 = Cell() - c4 = Cell() + c1 = XCell() + c2 = XCell() + c3 = XCell() + c4 = XCell() for a in (c1,c2,c3,c4): for b in (c1,c2,c3,c4): if a is not b: @@ -75,17 +25,18 @@ self.assert_(a.is_shared(b)) self.assertEquals(a, b) + def test_constant(self): + self.assertEquals(XConstant(5), XConstant(5)) + self.failIfEqual(XConstant(5), XConstant(6)) + self.failIfEqual(XConstant(5), XCell()) + -class TestAnnotationSet(test.IntTestCase): +class TestAnnotationHeap(test.IntTestCase): def setUp(self): - self.v1 = Variable('v1') - self.v2 = Variable('v2') - self.v3 = Variable('v3') - self.k1 = Constant(102938) - self.k2 = Constant('foobar') - self.k3 = Constant(-2) - self.k4 = Constant(102938) + self.c1 = XCell() + self.c2 = XCell() + self.c3 = XConstant(-2) def assertSameSet(self, a, b): a = list(a) @@ -101,17 +52,13 @@ a[i], a[j] = a[j], a[i] self.assertEquals(a, b) - def test_init(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v2], self.v3)] - a = AnnotationSet(lst) - self.assertSameSet(a, lst) - def test_add(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v2], self.v3)] - a = AnnotationSet() + lst = [SpaceOperation('add', [self.c1, self.c3], self.c2), + SpaceOperation('neg', [self.c2], self.c3)] + a = AnnotationHeap() + self.assertSameSet(a, []) a.add(lst[1]) + self.assertSameSet(a, [lst[1]]) a.add(lst[0]) self.assertSameSet(a, lst) a.add(lst[0]) @@ -119,118 +66,156 @@ a.add(lst[1]) self.assertSameSet(a, lst) - def test_add2(self): - c1 = Cell() - c2 = Cell() - c3 = Cell() - c4 = Cell() - c5 = Cell() - c6 = Cell() - c7 = Cell() - c8 = Cell() - c9 = Cell() - a = AnnotationSet() - op = SpaceOperation('add', [c1, c2], c3) - a.add(op) - self.assertSameSet(a, [op]) - op = SpaceOperation('add', [c4, self.k1], c5) - a.add(op) - self.assertSameSet(a, [op]) - op = SpaceOperation('add', [c6, self.k4], self.v3) - a.add(op) - self.assertSameSet(a, [op]) - op = SpaceOperation('add', [self.v1, self.k1], self.v3) - a.add(op) - self.assertSameSet(a, [op]) - - a.add(SpaceOperation('add', [self.v1, c7], self.v3)) - self.assertSameSet(a, [op]) - a.add(SpaceOperation('add', [self.v1, c9], c8)) - self.assertSameSet(a, [op]) - - def test_match1(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v1], self.k2), - SpaceOperation('neg', [self.v2], self.v3)] - a = AnnotationSet(lst) - for ann in lst: - self.assert_(a.match(ann)) - c = Cell() - self.assert_(a.match(SpaceOperation('add', [self.v1, self.k4], c))) - self.assertEquals(c.content, self.v2) - c = Cell() - c2 = Cell() - self.assert_(a.match(SpaceOperation('add', [self.v1, c], c2))) - self.assertEquals(c.content, self.k1) - self.assertEquals(c2.content, self.v2) - c = Cell() - self.assert_(a.match(SpaceOperation('neg', [c], self.v3))) - self.assertEquals(c.content, self.v2) - c = Cell() - self.failIf(a.match(SpaceOperation('add', [self.v2, self.k1], self.v2))) - self.failIf(a.match(SpaceOperation('add', [self.v2, self.k1], c))) - - def test_match2(self): - c1 = Cell() - c2 = Cell() - c3 = Cell() - c4 = Cell() - c4.share(c3) - c1.set(self.k4) - lst = [SpaceOperation('add', [self.v1, c1], self.v2), - SpaceOperation('neg', [self.v1], c2), - SpaceOperation('neg', [self.v2], c4)] - a = AnnotationSet(lst) - self.assert_(a.match(SpaceOperation('add', [self.v1, self.k1], self.v2))) - c = Cell() - self.assert_(a.match(SpaceOperation('add', [self.v1, self.k1], c))) - self.assertEquals(c.content, self.v2) - c = Cell() - self.assert_(a.match(SpaceOperation('neg', [self.v2], c))) - self.assert_(c.is_shared(c4)) - self.assert_(c.is_shared(c3)) - self.assert_(not c.is_shared(c2)) - self.assert_(not c.is_shared(c1)) - - self.assertEquals(c1.content, self.k1) - self.assertEquals(c2.content, None) - self.assertEquals(c3.content, None) - self.assertEquals(c4.content, None) - def test_enumerate(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v1], self.k2), - SpaceOperation('neg', [self.v2], self.v3)] - a = AnnotationSet(lst) - self.assertSameSet(list(a.enumerate()), lst) - - def test_renaming(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v1], self.k2), - SpaceOperation('neg', [self.v2], self.v3)] - renaming = {self.v1: [self.v3], - self.v2: [self.v2, self.v1]} - lst2 = [SpaceOperation('add', [self.v3, self.k1], self.v2), - SpaceOperation('add', [self.v3, self.k1], self.v1), - SpaceOperation('neg', [self.v3], self.k2)] - a = AnnotationSet(lst) - self.assertSameSet(list(a.enumerate(renaming)), lst2) - - def test_intersect(self): - lst = [SpaceOperation('type', [self.v2], self.k1), - SpaceOperation('type', [self.v1], self.k1), - SpaceOperation('type', [self.v3], self.k1), - ] - lst2 = [SpaceOperation('type', [self.v2], self.k4), - SpaceOperation('type', [self.v1], self.k4), - SpaceOperation('add', [self.v1, self.v2], self.v3), - ] - lst3 = [SpaceOperation('type', [self.v2], self.k1), - SpaceOperation('type', [self.v1], self.k1), - ] - a = AnnotationSet(lst) - a.intersect(AnnotationSet(lst2)) - self.assertSameSet(a, lst3) + lst = [SpaceOperation('add', [self.c1, self.c3], self.c2)] + a = AnnotationHeap(lst) + self.assertSameSet(a.enumerate(), lst) + + def test_get_opresult(self): + lst = [SpaceOperation('add', [self.c1, self.c3], self.c2)] + a = AnnotationHeap(lst) + self.assertEquals(a.get_opresult('add', [self.c1, self.c3]), self.c2) + self.assertEquals(a.get_opresult('add', [self.c1, self.c2]), None) + self.assertEquals(a.get_opresult('sub', [self.c1, self.c3]), None) + + def test_get_type(self): + lst = [SpaceOperation('type', [self.c1], self.c3), + SpaceOperation('type', [self.c2], self.c1)] + a = AnnotationHeap(lst) + self.assertEquals(a.get_type(self.c1), -2) + self.assertEquals(a.get_type(self.c2), None) + self.assertEquals(a.get_type(self.c3), None) + + def test_set_type(self): + a = AnnotationHeap() + a.set_type(self.c1, int) + lst = [SpaceOperation('type', [self.c1], XConstant(int))] + self.assertSameSet(a, lst) + + def test_merge_nothingyet(self): + lst = [SpaceOperation('add', [self.c1, self.c3], self.c2), + SpaceOperation('neg', [self.c2], self.c3)] + a = AnnotationHeap(lst) + # (c3) inter (all annotations) == (c3) + c, changeflag = a.merge(self.c3, nothingyet) + self.failIf(changeflag) + self.assertEquals(c, self.c3) + self.failIfEqual(c, nothingyet) + self.assertSameSet(a, lst) + + def test_merge_mutable1(self): + lst = [SpaceOperation('type', [self.c1], self.c3), + SpaceOperation('type', [self.c2], self.c3), + SpaceOperation('somethingelse', [self.c2, self.c3], self.c3)] + a = AnnotationHeap(lst) + # (c1) inter (c2) == (c1 shared with c2) + c, changeflag = a.merge(self.c1, self.c2) + self.failIf(changeflag) + self.assertEquals(c, self.c1) + self.assertEquals(c, self.c2) + self.assertEquals(self.c1, self.c2) + self.assertSameSet(a, [SpaceOperation('type', [c], self.c3)]) + + def test_merge_mutable2(self): + lst = [SpaceOperation('type', [self.c1], self.c3), + SpaceOperation('type', [self.c2], self.c3), + SpaceOperation('somethingelse', [self.c1, self.c3], self.c3)] + a = AnnotationHeap(lst) + # (c1) inter (c2) == (c1 shared with c2) + c, changeflag = a.merge(self.c1, self.c2) + self.assert_(changeflag) + self.assertEquals(c, self.c1) + self.assertEquals(c, self.c2) + self.assertEquals(self.c1, self.c2) + self.assertSameSet(a, [SpaceOperation('type', [c], self.c3)]) + + def test_merge_immutable(self): + lst = [SpaceOperation('type', [self.c1], self.c3), + SpaceOperation('type', [self.c2], self.c3), + SpaceOperation('immutable', [], self.c1), + SpaceOperation('immutable', [], self.c2), + SpaceOperation('somethingelse', [self.c2, self.c3], self.c3)] + a = AnnotationHeap(lst) + # (c1) inter (c2) == (some new c4) + c, changeflag = a.merge(self.c1, self.c2) + self.failIf(changeflag) # because only c2 has annotations dropped + self.failIfEqual(self.c1, self.c2) + # c could be equal to c1 here, but we don't require that + for op in [SpaceOperation('type', [c], self.c3), + SpaceOperation('immutable', [], c)]: + if op not in lst: + lst.append(op) + self.assertSameSet(a, lst) + + def test_merge_mutable_ex(self): + lst = [SpaceOperation('add', [self.c1, self.c2], self.c2), + SpaceOperation('neg', [self.c2], self.c1), + SpaceOperation('add', [self.c3, self.c2], self.c2), + SpaceOperation('immutable', [], self.c2)] + a = AnnotationHeap(lst) + # (c1) inter (c3) == (c1 shared with c3) + c, changeflag = a.merge(self.c1, self.c3) + self.assert_(changeflag) + self.assertEquals(c, self.c1) + self.assertEquals(c, self.c3) + self.assertEquals(self.c1, self.c3) + self.assertSameSet(a, [lst[0], lst[3]]) + self.assertSameSet(a, [lst[2], lst[3]]) + + def test_merge_immutable_ex(self): + lst = [SpaceOperation('add', [self.c1, self.c2], self.c2), + SpaceOperation('neg', [self.c2], self.c1), + SpaceOperation('add', [self.c3, self.c2], self.c2), + SpaceOperation('immutable', [], self.c1), + SpaceOperation('immutable', [], self.c2), + SpaceOperation('immutable', [], self.c3)] + a = AnnotationHeap(lst) + # (c1) inter (c3) == (some new c4) + c, changeflag = a.merge(self.c1, self.c3) + self.assert_(changeflag) # because 'neg(..)=c1' is dropped + self.failIfEqual(c, self.c1) + self.failIfEqual(c, self.c3) + lst += [SpaceOperation('add', [c, self.c2], self.c2), + SpaceOperation('immutable', [], c)] + self.assertSameSet(a, lst) + + def dont_test_merge_mutable_ex(self): + # This test is expected to fail at this point because the algorithms + # are not 100% theoretically correct, but probably quite good and + # clear enough right now. In theory in the intersection below + # 'add' should be kept. In practice the extra 'c3' messes things + # up. I can only think about much-more-obscure algos to fix that. + lst = [SpaceOperation('add', [self.c1, self.c3], self.c2), + SpaceOperation('neg', [self.c2], self.c1), + SpaceOperation('add', [self.c3, self.c3], self.c2), + SpaceOperation('immutable', [], self.c2)] + a = AnnotationHeap(lst) + # (c1) inter (c3) == (c1 shared with c3) + c, changeflag = a.merge(self.c1, self.c3) + self.assert_(changeflag) + self.assertEquals(c, self.c1) + self.assertEquals(c, self.c3) + self.assertEquals(self.c1, self.c3) + self.assertSameSet(a, [lst[0], lst[3]]) + self.assertSameSet(a, [lst[2], lst[3]]) + + def dont_test_merge_immutable_ex(self): + # Disabled -- same as above. + lst = [SpaceOperation('add', [self.c1, self.c3], self.c2), + SpaceOperation('neg', [self.c2], self.c1), + SpaceOperation('add', [self.c3, self.c3], self.c2), + SpaceOperation('immutable', [], self.c1), + SpaceOperation('immutable', [], self.c2), + SpaceOperation('immutable', [], self.c3)] + a = AnnotationHeap(lst) + # (c1) inter (c3) == (some new c4) + c, changeflag = a.merge(self.c1, self.c3) + self.assert_(changeflag) # because 'neg(..)=c1' is dropped + self.failIfEqual(c, self.c1) + self.failIfEqual(c, self.c3) + lst += [SpaceOperation('add', [c, self.c3], self.c2), + SpaceOperation('immutable', [], c)] + self.assertSameSet(a, lst) if __name__ == '__main__': From arigo at codespeak.net Mon Nov 17 16:21:35 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 17 Nov 2003 16:21:35 +0100 (MET) Subject: [pypy-svn] rev 2205 - in pypy/trunk/src/pypy/translator: . test Message-ID: <20031117152135.370E15BE89@thoth.codespeak.net> Author: arigo Date: Mon Nov 17 16:21:34 2003 New Revision: 2205 Added: pypy/trunk/src/pypy/translator/annrpython.py (props changed) - copied unchanged from rev 2203, pypy/trunk/src/pypy/translator/annotation.py pypy/trunk/src/pypy/translator/test/test_annrpython.py (contents, props changed) - copied, changed from rev 2203, pypy/trunk/src/pypy/translator/test/test_annotation.py Removed: pypy/trunk/src/pypy/translator/annotation.py pypy/trunk/src/pypy/translator/test/test_annotation.py Modified: pypy/trunk/src/pypy/translator/gencl.py pypy/trunk/src/pypy/translator/genpyrex.py pypy/trunk/src/pypy/translator/transform.py Log: Renaming annotation.py to annrpython.py because I want to use annotation.py for the Annotation class, whereas annrpython contains the RPython-specific rules. Deleted: /pypy/trunk/src/pypy/translator/annotation.py ============================================================================== --- /pypy/trunk/src/pypy/translator/annotation.py Mon Nov 17 16:21:34 2003 +++ (empty file) @@ -1,207 +0,0 @@ -from __future__ import generators - -from pypy.translator.annset import AnnotationSet, Cell, deref -from pypy.objspace.flow.model import Variable, Constant, SpaceOperation - -#class GraphGlobalVariable(Variable): -# pass - -class Annotator: - - def __init__(self, flowgraph): - self.flowgraph = flowgraph - - def build_types(self, input_arg_types): - input_ann = AnnotationSet() - for arg, arg_type in zip(self.flowgraph.getargs(), input_arg_types): - input_ann.set_type(arg, arg_type) - self.build_annotations(input_ann) - - def build_annotations(self,input_annotations): - self.annotated = {} - self.flowin(self.flowgraph.startblock,input_annotations) - - def get_return_value(self): - "Return the return_value variable." - return self.flowgraph.returnblock.inputargs[0] - - def get_variables_ann(self): - """Return a dict {Variable(): AnnotationSet()} mapping each variable - of the control flow graph to a set of annotations that apply to it.""" - # XXX this assumes that all variables are local to a single block, - # and returns for each variable the annotations for that block. - # This assumption is clearly false because of the EggBlocks. - # This has to be fixed anyway. - result = {} - for block, ann in self.annotated.items(): - for v in block.getvariables(): - #XXX assert v not in result, "Variables must not be shared" - result[v] = ann - return result - - def simplify_calls(self): - for block, ann in self.annotated.iteritems(): - newops = [] - for op in block.operations: - if op.opname == "call": - w_func, w_varargs, w_kwargs = op.args - c = Cell() - ann.match(SpaceOperation('len', [w_varargs], c)) - if isinstance(c.content, Constant): - length = c.content.value - args_w = [w_func] - for i in range(length): - c = Cell() - if not ann.match(SpaceOperation('getitem', [ - w_varargs, Constant(i)], c)): - break - args_w.append(deref(c)) - else: - op = SpaceOperation('simple_call', args_w, op.result) - # XXX check that w_kwargs is empty - newops.append(op) - block.operations = newops - - def simplify(self): - self.simplify_calls() - - #__________________________________________________ - - def flowin(self, block, annotations): - if block not in self.annotated: - oldlen = None - self.annotated[block] = blockannotations = annotations - else: - blockannotations = self.annotated[block] - oldlen = len(blockannotations) - #import sys; print >> sys.stderr, block, blockannotations - #import sys; print >> sys.stderr, '/\\', annotations, '==>', - blockannotations.intersect(annotations) - #import sys; print >> sys.stderr, blockannotations - - for op in block.operations: - self.consider_op(op, blockannotations) - # assert monotonic decrease - assert (oldlen is None or len(blockannotations) <= oldlen), ( - block, oldlen, blockannotations) - if len(blockannotations) != oldlen: - for link in block.exits: - self.flownext(link,block) - - def consider_op(self,op,annotations): - consider_meth = getattr(self,'consider_op_'+op.opname,None) - if consider_meth is not None: - consider_meth(op,annotations) - - def consider_op_add(self,op,annotations): - arg1,arg2 = op.args - type1 = annotations.get_type(arg1) - type2 = annotations.get_type(arg2) - if type1 is int and type2 is int: - annotations.set_type(op.result, int) - elif type1 in (int, long) and type2 in (int, long): - annotations.set_type(op.result, long) - if type1 is str and type2 is str: - annotations.set_type(op.result, str) - if type1 is list and type2 is list: - annotations.set_type(op.result, list) - - consider_op_inplace_add = consider_op_add - - def consider_op_sub(self, op, annotations): - arg1, arg2 = op.args - type1 = annotations.get_type(arg1) - type2 = annotations.get_type(arg2) - if type1 is int and type2 is int: - annotations.set_type(op.result, int) - elif type1 in (int, long) and type2 in (int, long): - annotations.set_type(op.result, long) - - consider_op_and_ = consider_op_sub # trailing underline - consider_op_inplace_lshift = consider_op_sub - - def consider_op_is_true(self, op, annotations): - annotations.set_type(op.result, bool) - - consider_op_not_ = consider_op_is_true - - def consider_op_lt(self, op, annotations): - annotations.set_type(op.result, bool) - - consider_op_le = consider_op_lt - consider_op_eq = consider_op_lt - consider_op_ne = consider_op_lt - consider_op_gt = consider_op_lt - consider_op_ge = consider_op_lt - - def consider_op_newtuple(self,op,annotations): - annotations.set_type(op.result,tuple) - ann = SpaceOperation("len",[op.result],Constant(len(op.args))) - annotations.add(ann) - for i in range(len(op.args)): - ann = SpaceOperation("getitem",[op.result,Constant(i)],op.args[i]) - annotations.add(ann) - - def consider_op_newlist(self, op, annotations): - annotations.set_type(op.result, list) - - def consider_op_newslice(self,op,annotations): - annotations.set_type(op.result, slice) - - def consider_op_getitem(self, op, annotations): - arg1,arg2 = op.args - type1 = annotations.get_type(arg1) - type2 = annotations.get_type(arg2) - if type1 in (list, tuple) and type2 is slice: - annotations.set_type(op.result, type1) - - def consider_op_call(self, op, annotations): - func = op.args[0] - if not isinstance(func, Constant): - return - func = func.value - # XXX: generalize this later - if func is range: - annotations.set_type(op.result, list) - if func is pow: - varargs = op.args[1] - def getitem(var, i): - class NoMatch(Exception): pass - c = Cell() - match = annotations.match( - SpaceOperation('getitem', (var, Constant(i)), c)) - if match: return deref(c) - else: raise NoMatch - try: - tp1 = annotations.get_type(getitem(varargs, 0)) - tp2 = annotations.get_type(getitem(varargs, 1)) - if tp1 is int and tp2 is int: - annotations.set_type(op.result, int) - except NoMatch: - pass - - def consider_const(self,to_var,const,annotations): - if getattr(const, 'dummy', False): - return # undefined local variables - annotations.set_type(to_var,type(const.value)) - if isinstance(const.value, list): - pass # XXX say something about the type of the elements - elif isinstance(const.value, tuple): - pass # XXX say something about the elements - - def flownext(self,link,curblock): - renaming = {} - newannotations = AnnotationSet() - - for w_from,w_to in zip(link.args,link.target.inputargs): - if isinstance(w_from,Variable): - renaming.setdefault(w_from, []).append(w_to) - else: - self.consider_const(w_to,w_from,newannotations) - - #import sys; print >> sys.stderr, self.annotated[curblock] - #import sys; print >> sys.stderr, renaming - for ann in self.annotated[curblock].enumerate(renaming): - newannotations.add(ann) - #import sys; print >> sys.stderr, newannotations - self.flowin(link.target,newannotations) Modified: pypy/trunk/src/pypy/translator/gencl.py ============================================================================== --- pypy/trunk/src/pypy/translator/gencl.py (original) +++ pypy/trunk/src/pypy/translator/gencl.py Mon Nov 17 16:21:34 2003 @@ -1,6 +1,6 @@ import autopath from pypy.objspace.flow.model import * -from pypy.translator.annotation import Annotator +from pypy.translator.annrpython import Annotator from pypy.translator.simplify import simplify_graph from pypy.translator.transform import transform_graph Modified: pypy/trunk/src/pypy/translator/genpyrex.py ============================================================================== --- pypy/trunk/src/pypy/translator/genpyrex.py (original) +++ pypy/trunk/src/pypy/translator/genpyrex.py Mon Nov 17 16:21:34 2003 @@ -5,7 +5,7 @@ from pypy.interpreter.baseobjspace import ObjSpace from pypy.objspace.flow.model import Variable, Constant, SpaceOperation from pypy.objspace.flow.model import mkentrymap -from pypy.translator.annotation import Annotator +from pypy.translator.annrpython import Annotator class Op: def __init__(self, operation, gen, block): Deleted: /pypy/trunk/src/pypy/translator/test/test_annotation.py ============================================================================== --- /pypy/trunk/src/pypy/translator/test/test_annotation.py Mon Nov 17 16:21:34 2003 +++ (empty file) @@ -1,135 +0,0 @@ - -import autopath -from pypy.tool import test -from pypy.tool.udir import udir - -from pypy.translator.annotation import Annotator -from pypy.objspace.flow.model import * - -class AnnonateTestCase(test.IntTestCase): - def setUp(self): - self.space = test.objspace('flow') - - def make_ann(self, func): - """ make a pyrex-generated cfunction from the given func """ - import inspect - try: - func = func.im_func - except AttributeError: - pass - name = func.func_name - funcgraph = self.space.build_flow(func) - funcgraph.source = inspect.getsource(func) - return Annotator(funcgraph) - - def reallyshow(self, graph): - import os - from pypy.translator.test.make_dot import make_dot - from pypy.tool.udir import udir - dest = make_dot(graph, udir, 'ps') - os.system('gv %s' % str(dest)) - - def test_simple_func(self): - """ - one test source: - def f(x): - return x+1 - """ - x = Variable("x") - result = Variable("result") - op = SpaceOperation("add", [x, Constant(1)], result) - block = Block([x]) - fun = FunctionGraph("f", block) - block.operations.append(op) - block.closeblock(Link([result], fun.returnblock)) - a = Annotator(fun) - a.build_types([int]) - end_var = a.get_return_value() - end_ann = a.get_variables_ann()[end_var] - self.assertEquals(end_ann.get_type(end_var), int) - - def test_while(self): - """ - one test source: - def f(i): - while i > 0: - i = i - 1 - return i - """ - i = Variable("i") - conditionres = Variable("conditionres") - conditionop = SpaceOperation("gt", [i, Constant(0)], conditionres) - decop = SpaceOperation("add", [i, Constant(-1)], i) - headerblock = Block([i]) - whileblock = Block([i]) - - fun = FunctionGraph("f", headerblock) - headerblock.operations.append(conditionop) - headerblock.exitswitch = conditionres - headerblock.closeblock(Link([i], fun.returnblock, False), - Link([i], whileblock, True)) - whileblock.operations.append(decop) - whileblock.closeblock(Link([i], headerblock)) - - a = Annotator(fun) - a.build_types([int]) - end_var = a.get_return_value() - end_ann = a.get_variables_ann()[end_var] - self.assertEquals(end_ann.get_type(end_var), int) - - def test_while_sum(self): - """ - one test source: - def f(i): - sum = 0 - while i > 0: - sum = sum + i - i = i - 1 - return sum - """ - i = Variable("i") - sum = Variable("sum") - - conditionres = Variable("conditionres") - conditionop = SpaceOperation("gt", [i, Constant(0)], conditionres) - decop = SpaceOperation("add", [i, Constant(-1)], i) - addop = SpaceOperation("add", [i, sum], sum) - startblock = Block([i]) - headerblock = Block([i, sum]) - whileblock = Block([i, sum]) - - fun = FunctionGraph("f", startblock) - startblock.closeblock(Link([i, Constant(0)], headerblock)) - headerblock.operations.append(conditionop) - headerblock.exitswitch = conditionres - headerblock.closeblock(Link([sum], fun.returnblock, False), - Link([i, sum], whileblock, True)) - whileblock.operations.append(addop) - whileblock.operations.append(decop) - whileblock.closeblock(Link([i, sum], headerblock)) - - a = Annotator(fun) - #import sys; print >> sys.stderr, a.build_annotations(input_ann) - a.build_types([int]) - end_var = a.get_return_value() - end_ann = a.get_variables_ann()[end_var] - self.assertEquals(end_ann.get_type(end_var), int) - - def test_simplify_calls(self): - a = self.make_ann(f_calls_g) - a.build_types([int]) - a.simplify_calls() - #self.reallyshow(a.flowgraph) - -def g(n): - return [0,1,2,n] - -def f_calls_g(n): - total = 0 - for i in g(n): - total += i - return total - - -if __name__ == '__main__': - test.main() Copied: pypy/trunk/src/pypy/translator/test/test_annrpython.py (from rev 2203, pypy/trunk/src/pypy/translator/test/test_annotation.py) ============================================================================== --- pypy/trunk/src/pypy/translator/test/test_annotation.py (original) +++ pypy/trunk/src/pypy/translator/test/test_annrpython.py Mon Nov 17 16:21:34 2003 @@ -3,7 +3,7 @@ from pypy.tool import test from pypy.tool.udir import udir -from pypy.translator.annotation import Annotator +from pypy.translator.annrpython import Annotator from pypy.objspace.flow.model import * class AnnonateTestCase(test.IntTestCase): Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Mon Nov 17 16:21:34 2003 @@ -6,7 +6,7 @@ import autopath from pypy.objspace.flow.model import Variable, Constant, SpaceOperation -from pypy.translator.annotation import Annotator +from pypy.translator.annrpython import Annotator # b = newlist(a) # d = mul(b, int c) From arigo at codespeak.net Mon Nov 17 18:29:21 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 17 Nov 2003 18:29:21 +0100 (MET) Subject: [pypy-svn] rev 2206 - in pypy/trunk/src/pypy/translator: . test Message-ID: <20031117172921.A7FA15A411@thoth.codespeak.net> Author: arigo Date: Mon Nov 17 18:29:19 2003 New Revision: 2206 Added: pypy/trunk/src/pypy/translator/annotation.py pypy/trunk/src/pypy/translator/test/test_annotation.py Modified: pypy/trunk/src/pypy/translator/annheap.py pypy/trunk/src/pypy/translator/test/test_annheap.py Log: Added dependencies among annotations. AnnotationHeap.merge() will now recursively kill annotations and those that depended on them. The new Transaction class is needed to record dependencies. Annotations are no longer looked up through the AnnotationHeap but via a Transaction instance, which records it so that new annotations can be marked as depending on the existing annotations that have just been looked up. Modified: pypy/trunk/src/pypy/translator/annheap.py ============================================================================== --- pypy/trunk/src/pypy/translator/annheap.py (original) +++ pypy/trunk/src/pypy/translator/annheap.py Mon Nov 17 18:29:19 2003 @@ -1,152 +1,72 @@ from __future__ import generators -import weakref -from pypy.objspace.flow.model import Variable, Constant, SpaceOperation - - -class XCell: - """A placeholder for a heap object contained in an AnnotationHeap. - It represents an object that will actually appear at run-time in the heap. - XCells are the arguments and return value of SpaceOperations when - used as annotations.""" - - counter = 0 - - # Multiple XCells can be "shared"; a group of shared cells - # act essentially like a single cell (they become all equal). - - def __init__(self, name=None): - if not name: - name = 'X%d' % XCell.counter - XCell.counter += 1 - self.name = name - self.shared = [] # list of weakrefs to XCells - # defining a group of shared cells. - - def __repr__(self): - names = [cell.name for cell in self.cellsingroup()] - names.sort() - return '=='.join(names) - - def __eq__(self, other): - "Two sharing cells are equal." - return isinstance(other, XCell) and self.is_shared(other) - - def __ne__(self, other): - return not (self == other) - - def cellsingroup(self): - if self.shared: - l = [s() for s in self.shared] - assert self in l - return [c for c in l if c is not None] - else: - return [self] - - def getsharelist(self): - if not self.shared: - self.shared = [weakref.ref(self)] - return self.shared - - def is_shared(self, other): - "Test if two cells are shared." - return self.shared is other.shared - - def share(self, other): - "Make two cells shared." - if not self.is_shared(other): - lst1 = self.getsharelist() - lst2 = other.getsharelist() - for s in lst2: - c = s() - if c is not None: - c.shared = lst1 - lst1.append(s) - - -class XConstant(XCell): - """A fully determined XCell. For immutable constants.""" - - def __init__(self, value): - XCell.__init__(self) - self.value = value - - def __eq__(self, other): - "Two constants with the same value are equal." - return (isinstance(other, XConstant) and self.value == other.value - or XCell.__eq__(self, other)) - - -# The more annotations about an XCell, the least general -# it is. Extreme case: *all* possible annotations stand for an -# object that cannot exist (e.g. the return value of a function -# that never returns or we didn't see return so far). -# This is specified by using nothingyet instead of a real XCell(). -# Conversely, *no* annotation stands for any object. - -nothingyet = XCell('nothingyet') +from annotation import Annotation, XCell, XConstant, nothingyet class AnnotationHeap: - """An annotation heap is a (large) family of annotations. - An annotation is a SpaceOperation with XCells as arguments and result.""" + """An annotation heap is a (large) family of Annotations.""" + + # XXX STORED AS A PLAIN LIST, THE COMPLEXITY IS PLAINLY WRONG def __init__(self, annlist=[]): - self.annlist = [] # List of annotations XXX optimize - for ann in annlist: - self.add(ann) + self.annlist = list(annlist) # List of annotations def dump(self): # debugging for ann in self.enumerate(): print ann - def add(self, annotation): - """Register an annotation into the heap.""" - if annotation not in self.annlist: - self.annlist.append(annotation) - def enumerate(self): """Enumerates all annotations in the heap.""" return iter(self.annlist) __iter__ = enumerate - def set_type(self, cell, type): - """Register an annotation describing the type of the object 'cell'.""" - self.add(SpaceOperation('type', [cell], XConstant(type))) + def simplify(self, kill=[]): + """Kill annotations in the list, and recursively all the annotations + that depend on them, and simplify the resulting heap to remove + duplicates.""" + # temporarykey() returns a tuple with all the information about + # the annotation; equal temporarykey() means equal annotations. + # Such keys are temporary because making new XCells shared can + # change the temporarykey(), but this doesn't occur during + # one call to simplify(). + + allkeys = {} # map temporarykeys to Annotation instances + for ann in self.annlist: + key = ann.temporarykey() + if key in allkeys: # duplicate? + previous = allkeys[key] + previous.forward_deps += ann.forward_deps # merge + else: + allkeys[key] = ann - def get_type(self, cell): - """Get the type of 'cell', as specified by the annotations, or None.""" - c = self.get_opresult('type', [cell]) - if isinstance(c, XConstant): - return c.value - else: - return None + killkeys = {} # set of temporarykeys of annotations to remove + for ann in kill: + killkeys[ann.temporarykey()] = True + + pending = killkeys.keys() + for key in pending: + if key in allkeys: + ann = allkeys[key] + del allkeys[key] # remove annotations from the dict + for dep in ann.forward_deps: # propagate dependencies + depkey = dep.temporarykey() + if depkey not in killkeys: + killkeys[depkey] = True + pending.append(depkey) - def get_opresult(self, name, args): - """Return the Cell with the annotation 'name(args) = cell', - or None if there is no such annotation, or several different ones.""" - result = None - for ann in self.annlist: - if ann.opname == name and ann.args == args: - if result is None: - result = ann.result - elif ann.result != result: - return None - return result + self.annlist = allkeys.values() def merge(self, oldcell, newcell): """Update the heap to account for the merging of oldcell and newcell. - Return (resultcell, changeflag) where resultcell is the merged cell. - changeflag is false only if the merged cell is equal to oldcell and - no annotations about oldcell have been dropped.""" + Return the merged cell.""" if newcell is nothingyet or newcell == oldcell: - return oldcell, False + return oldcell elif oldcell is nothingyet: - return newcell, True + return newcell else: # find the annotations common to oldcell and newcell common = [] - deleting = False # means deleting an annotation about oldcell + deleting = [] # annotations about oldcell that must be killed for ann in self.annlist: if oldcell in ann.args or oldcell == ann.result: test1 = rename(ann, oldcell, newcell) @@ -154,30 +74,32 @@ if test1 in self.annlist and test2 in self.annlist: common.append(test1) else: - deleting = True + deleting.append(test1) # the involved objects are immutable if we have both # 'immutable() -> oldcell' and 'immutable() -> newcell' - if SpaceOperation('immutable', [], newcell) in common: + if Annotation('immutable', [], newcell) in common: # for immutable objects we can create a new cell if necessary if not deleting: - return oldcell, False # nothing must be removed from oldcell + return oldcell # nothing must be removed from oldcell else: resultcell = XCell() # invent a new cell for ann in common: - self.add(rename(ann, newcell, resultcell)) - return resultcell, True + self.annlist.append(rename(ann, newcell, resultcell)) + return resultcell else: # for mutable objects we must identify oldcell and newcell, # and only keep the common annotations newcell.share(oldcell) - # add to 'common' all annotations that don't talk about oldcell - # (nor newcell, but at this point oldcell == newcell) + # search again and list all annotations that talk about + # oldcell or newcell (same thing now) but are not in 'common' + deleting = [] for ann in self.annlist: - if not (oldcell in ann.args or oldcell == ann.result): - common.append(ann) - # 'common' is now the list of remaining annotations - self.annlist[:] = common - return oldcell, deleting + if oldcell in ann.args or oldcell == ann.result: + if ann not in common: + deleting.append(ann) + # apply changes + self.simplify(kill=deleting) + return oldcell def rename(ann, oldcell, newcell): @@ -190,4 +112,57 @@ a = ann.result if a == oldcell: a = newcell - return SpaceOperation(ann.opname, args, a) + return Annotation(ann.opname, args, a) + + +class Transaction: + """A transaction contains methods to look for annotations in the + AnnotationHeap and create new annotations accordingly. Each + Transaction instance records which Annotations were needed, which + allows dependencies to be tracked.""" + + def __init__(self, heap): + self.heap = heap + self.using_annotations = [] # annotations that we have used + + def get(self, opname, args): + """Return the Cell with the annotation 'opname(args) -> Cell', + or None if there is no such annotation or several different ones.""" + matchann = None + for ann in self.heap.annlist: + if ann.opname == opname and ann.args == args: + if matchann is None: + matchann = ann # first possible annotation + elif matchann != ann: + return None # more than one annotation would match + if matchann is None: + return None + else: + self.using(matchann) + return matchann.result + # a note about duplicate Annotations in annlist: their forward_deps + # lists will automatically be merged during the next simplify(), + # so that we only need to record the dependency from one of them. + + def set(self, opname, args, result): + """Put a new annotation into the AnnotationHeap.""" + ann = Annotation(opname, args, result) + for prev in self.using_annotations: + prev.forward_deps.append(ann) + self.heap.annlist.append(ann) + + def get_type(self, cell): + """Get the type of 'cell', as specified by the annotations, or None.""" + c = self.get('type', [cell]) + if isinstance(c, XConstant): + return c.value + else: + return None + + def set_type(self, cell, type): + """Register an annotation describing the type of the object 'cell'.""" + self.set('type', [cell], XConstant(type)) + + def using(self, ann): + """Mark 'ann' as used in this transaction.""" + self.using_annotations.append(ann) Added: pypy/trunk/src/pypy/translator/annotation.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/translator/annotation.py Mon Nov 17 18:29:19 2003 @@ -0,0 +1,123 @@ +import weakref + + +class Annotation: + """An Annotation asserts something about heap objects represented + by XCell instances.""" + + # Note that this is very much like a SpaceOperation, but we keep + # them separate because they have different purposes. + + # Attention, handle Annotations with care! Two Annotations that + # were initially different could become equal when XCells become + # shared. This is the reason why Annotations are not hashable. + + def __init__(self, opname, args, result): + self.opname = opname # operation name + self.args = list(args) # list of XCells + self.result = result # an XCell + self.forward_deps = [] # annotations that depend on this one + + def __eq__(self, other): + return (self.__class__ is other.__class__ and + self.opname == other.opname and + self.args == other.args and + self.result == other.result) + + def __ne__(self, other): + return not (self == other) + + def __repr__(self): + return "%s(%s) -> %s" % (self.opname, ", ".join(map(repr, self.args)), + self.result) + + def temporarykey(self): + lst = [self.opname, self.result.temporarykey()] + lst += [arg.temporarykey() for arg in self.args] + return tuple(lst) + + +class XCell: + """A placeholder for a heap object contained in an AnnotationHeap. + It represents an object that will actually appear at run-time in the heap. + XCells are the arguments and return value of Annotations.""" + + counter = 0 + + # Multiple XCells can be "shared"; a group of shared cells + # act essentially like a single cell (they become all equal). + + def __init__(self, name=None): + if not name: + name = 'X%d' % XCell.counter + XCell.counter += 1 + self.name = name + self.shared = [] # list of weakrefs to XCells + # defining a group of shared cells. + + def __repr__(self): + names = [cell.name for cell in self.cellsingroup()] + names.sort() + return '=='.join(names) + + def __eq__(self, other): + "Two sharing cells are equal." + return isinstance(other, XCell) and self.is_shared(other) + + def __ne__(self, other): + return not (self == other) + + def temporarykey(self): + ids = [id(cell) for cell in self.cellsingroup()] + return min(ids) + + def cellsingroup(self): + if self.shared: + l = [s() for s in self.shared] + assert self in l + return [c for c in l if c is not None] + else: + return [self] + + def getsharelist(self): + if not self.shared: + self.shared = [weakref.ref(self)] + return self.shared + + def is_shared(self, other): + "Test if two cells are shared." + return self.shared is other.shared + + def share(self, other): + "Make two cells shared." + if not self.is_shared(other): + lst1 = self.getsharelist() + lst2 = other.getsharelist() + for s in lst2: + c = s() + if c is not None: + c.shared = lst1 + lst1.append(s) + + +class XConstant(XCell): + """A fully determined XCell. For immutable constants.""" + + def __init__(self, value): + XCell.__init__(self) + self.value = value + + def __eq__(self, other): + "Two constants with the same value are equal." + return (isinstance(other, XConstant) and self.value == other.value + or XCell.__eq__(self, other)) + + +# The more annotations about an XCell, the least general +# it is. Extreme case: *all* possible annotations stand for an +# object that cannot exist (e.g. the return value of a function +# that never returns or we didn't see return so far). +# This is specified by using nothingyet instead of a real XCell(). +# Conversely, *no* annotation stands for any object. + +nothingyet = XCell('nothingyet') Modified: pypy/trunk/src/pypy/translator/test/test_annheap.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/test_annheap.py (original) +++ pypy/trunk/src/pypy/translator/test/test_annheap.py Mon Nov 17 18:29:19 2003 @@ -2,33 +2,8 @@ import autopath from pypy.tool import test -from pypy.translator.annheap import XCell, XConstant, AnnotationHeap, nothingyet -from pypy.objspace.flow.model import SpaceOperation - - -class TestXCell(test.IntTestCase): - - def test_is_shared(self): - c1 = XCell() - c2 = XCell() - c3 = XCell() - c4 = XCell() - for a in (c1,c2,c3,c4): - for b in (c1,c2,c3,c4): - if a is not b: - self.failIfEqual(a, b) - c1.share(c2) - c4.share(c2) - c1.share(c3) - for a in (c1,c2,c3,c4): - for b in (c1,c2,c3,c4): - self.assert_(a.is_shared(b)) - self.assertEquals(a, b) - - def test_constant(self): - self.assertEquals(XConstant(5), XConstant(5)) - self.failIfEqual(XConstant(5), XConstant(6)) - self.failIfEqual(XConstant(5), XCell()) +from pypy.translator.annotation import XCell, XConstant, nothingyet, Annotation +from pypy.translator.annheap import AnnotationHeap, Transaction class TestAnnotationHeap(test.IntTestCase): @@ -52,110 +27,103 @@ a[i], a[j] = a[j], a[i] self.assertEquals(a, b) - def test_add(self): - lst = [SpaceOperation('add', [self.c1, self.c3], self.c2), - SpaceOperation('neg', [self.c2], self.c3)] - a = AnnotationHeap() - self.assertSameSet(a, []) - a.add(lst[1]) - self.assertSameSet(a, [lst[1]]) - a.add(lst[0]) - self.assertSameSet(a, lst) - a.add(lst[0]) - self.assertSameSet(a, lst) - a.add(lst[1]) - self.assertSameSet(a, lst) - def test_enumerate(self): - lst = [SpaceOperation('add', [self.c1, self.c3], self.c2)] + lst = [Annotation('add', [self.c1, self.c3], self.c2)] a = AnnotationHeap(lst) self.assertSameSet(a.enumerate(), lst) - def test_get_opresult(self): - lst = [SpaceOperation('add', [self.c1, self.c3], self.c2)] + def test_simplify(self): + lst = [Annotation('add', [self.c1, self.c3], self.c2), + Annotation('add', [self.c1, self.c2], self.c2), + Annotation('neg', [self.c2], self.c3)] + a = AnnotationHeap(lst) + a.simplify() + self.assertSameSet(a, lst) + + self.c2.share(self.c3) + a.simplify() + self.assertSameSet(a, lst[1:]) + + def test_simplify_kill(self): + ann1 = Annotation('add', [self.c1, self.c3], self.c2) + lst = [ann1, + Annotation('add', [self.c1, self.c2], self.c2), + Annotation('neg', [self.c2], self.c3)] + a = AnnotationHeap(lst) + a.simplify(kill=[ann1]) + self.assertSameSet(a, lst[1:]) + + def test_simplify_kill_deps(self): + ann1 = Annotation('add', [self.c1, self.c3], self.c2) + ann2 = Annotation('add', [self.c1, self.c2], self.c2) + ann3 = Annotation('add', [self.c1, self.c1], self.c2) + ann1.forward_deps.append(ann2) + ann2.forward_deps.append(ann3) + lst = [ann1, ann2, ann3, + Annotation('neg', [self.c2], self.c3)] a = AnnotationHeap(lst) - self.assertEquals(a.get_opresult('add', [self.c1, self.c3]), self.c2) - self.assertEquals(a.get_opresult('add', [self.c1, self.c2]), None) - self.assertEquals(a.get_opresult('sub', [self.c1, self.c3]), None) - - def test_get_type(self): - lst = [SpaceOperation('type', [self.c1], self.c3), - SpaceOperation('type', [self.c2], self.c1)] - a = AnnotationHeap(lst) - self.assertEquals(a.get_type(self.c1), -2) - self.assertEquals(a.get_type(self.c2), None) - self.assertEquals(a.get_type(self.c3), None) - - def test_set_type(self): - a = AnnotationHeap() - a.set_type(self.c1, int) - lst = [SpaceOperation('type', [self.c1], XConstant(int))] - self.assertSameSet(a, lst) + a.simplify(kill=[ann1]) + self.assertSameSet(a, lst[3:]) def test_merge_nothingyet(self): - lst = [SpaceOperation('add', [self.c1, self.c3], self.c2), - SpaceOperation('neg', [self.c2], self.c3)] + lst = [Annotation('add', [self.c1, self.c3], self.c2), + Annotation('neg', [self.c2], self.c3)] a = AnnotationHeap(lst) # (c3) inter (all annotations) == (c3) - c, changeflag = a.merge(self.c3, nothingyet) - self.failIf(changeflag) + c = a.merge(self.c3, nothingyet) self.assertEquals(c, self.c3) self.failIfEqual(c, nothingyet) self.assertSameSet(a, lst) def test_merge_mutable1(self): - lst = [SpaceOperation('type', [self.c1], self.c3), - SpaceOperation('type', [self.c2], self.c3), - SpaceOperation('somethingelse', [self.c2, self.c3], self.c3)] + lst = [Annotation('type', [self.c1], self.c3), + Annotation('type', [self.c2], self.c3), + Annotation('somethingelse', [self.c2, self.c3], self.c3)] a = AnnotationHeap(lst) # (c1) inter (c2) == (c1 shared with c2) - c, changeflag = a.merge(self.c1, self.c2) - self.failIf(changeflag) + c = a.merge(self.c1, self.c2) self.assertEquals(c, self.c1) self.assertEquals(c, self.c2) self.assertEquals(self.c1, self.c2) - self.assertSameSet(a, [SpaceOperation('type', [c], self.c3)]) + self.assertSameSet(a, [Annotation('type', [c], self.c3)]) def test_merge_mutable2(self): - lst = [SpaceOperation('type', [self.c1], self.c3), - SpaceOperation('type', [self.c2], self.c3), - SpaceOperation('somethingelse', [self.c1, self.c3], self.c3)] + lst = [Annotation('type', [self.c1], self.c3), + Annotation('type', [self.c2], self.c3), + Annotation('somethingelse', [self.c1, self.c3], self.c3)] a = AnnotationHeap(lst) # (c1) inter (c2) == (c1 shared with c2) - c, changeflag = a.merge(self.c1, self.c2) - self.assert_(changeflag) + c = a.merge(self.c1, self.c2) self.assertEquals(c, self.c1) self.assertEquals(c, self.c2) self.assertEquals(self.c1, self.c2) - self.assertSameSet(a, [SpaceOperation('type', [c], self.c3)]) + self.assertSameSet(a, [Annotation('type', [c], self.c3)]) def test_merge_immutable(self): - lst = [SpaceOperation('type', [self.c1], self.c3), - SpaceOperation('type', [self.c2], self.c3), - SpaceOperation('immutable', [], self.c1), - SpaceOperation('immutable', [], self.c2), - SpaceOperation('somethingelse', [self.c2, self.c3], self.c3)] + lst = [Annotation('type', [self.c1], self.c3), + Annotation('type', [self.c2], self.c3), + Annotation('immutable', [], self.c1), + Annotation('immutable', [], self.c2), + Annotation('somethingelse', [self.c2, self.c3], self.c3)] a = AnnotationHeap(lst) # (c1) inter (c2) == (some new c4) - c, changeflag = a.merge(self.c1, self.c2) - self.failIf(changeflag) # because only c2 has annotations dropped + c = a.merge(self.c1, self.c2) self.failIfEqual(self.c1, self.c2) # c could be equal to c1 here, but we don't require that - for op in [SpaceOperation('type', [c], self.c3), - SpaceOperation('immutable', [], c)]: + for op in [Annotation('type', [c], self.c3), + Annotation('immutable', [], c)]: if op not in lst: lst.append(op) self.assertSameSet(a, lst) def test_merge_mutable_ex(self): - lst = [SpaceOperation('add', [self.c1, self.c2], self.c2), - SpaceOperation('neg', [self.c2], self.c1), - SpaceOperation('add', [self.c3, self.c2], self.c2), - SpaceOperation('immutable', [], self.c2)] + lst = [Annotation('add', [self.c1, self.c2], self.c2), + Annotation('neg', [self.c2], self.c1), + Annotation('add', [self.c3, self.c2], self.c2), + Annotation('immutable', [], self.c2)] a = AnnotationHeap(lst) # (c1) inter (c3) == (c1 shared with c3) - c, changeflag = a.merge(self.c1, self.c3) - self.assert_(changeflag) + c = a.merge(self.c1, self.c3) self.assertEquals(c, self.c1) self.assertEquals(c, self.c3) self.assertEquals(self.c1, self.c3) @@ -163,20 +131,19 @@ self.assertSameSet(a, [lst[2], lst[3]]) def test_merge_immutable_ex(self): - lst = [SpaceOperation('add', [self.c1, self.c2], self.c2), - SpaceOperation('neg', [self.c2], self.c1), - SpaceOperation('add', [self.c3, self.c2], self.c2), - SpaceOperation('immutable', [], self.c1), - SpaceOperation('immutable', [], self.c2), - SpaceOperation('immutable', [], self.c3)] + lst = [Annotation('add', [self.c1, self.c2], self.c2), + Annotation('neg', [self.c2], self.c1), + Annotation('add', [self.c3, self.c2], self.c2), + Annotation('immutable', [], self.c1), + Annotation('immutable', [], self.c2), + Annotation('immutable', [], self.c3)] a = AnnotationHeap(lst) # (c1) inter (c3) == (some new c4) - c, changeflag = a.merge(self.c1, self.c3) - self.assert_(changeflag) # because 'neg(..)=c1' is dropped + c = a.merge(self.c1, self.c3) self.failIfEqual(c, self.c1) self.failIfEqual(c, self.c3) - lst += [SpaceOperation('add', [c, self.c2], self.c2), - SpaceOperation('immutable', [], c)] + lst += [Annotation('add', [c, self.c2], self.c2), + Annotation('immutable', [], c)] self.assertSameSet(a, lst) def dont_test_merge_mutable_ex(self): @@ -185,14 +152,13 @@ # clear enough right now. In theory in the intersection below # 'add' should be kept. In practice the extra 'c3' messes things # up. I can only think about much-more-obscure algos to fix that. - lst = [SpaceOperation('add', [self.c1, self.c3], self.c2), - SpaceOperation('neg', [self.c2], self.c1), - SpaceOperation('add', [self.c3, self.c3], self.c2), - SpaceOperation('immutable', [], self.c2)] + lst = [Annotation('add', [self.c1, self.c3], self.c2), + Annotation('neg', [self.c2], self.c1), + Annotation('add', [self.c3, self.c3], self.c2), + Annotation('immutable', [], self.c2)] a = AnnotationHeap(lst) # (c1) inter (c3) == (c1 shared with c3) - c, changeflag = a.merge(self.c1, self.c3) - self.assert_(changeflag) + c = a.merge(self.c1, self.c3) self.assertEquals(c, self.c1) self.assertEquals(c, self.c3) self.assertEquals(self.c1, self.c3) @@ -201,22 +167,77 @@ def dont_test_merge_immutable_ex(self): # Disabled -- same as above. - lst = [SpaceOperation('add', [self.c1, self.c3], self.c2), - SpaceOperation('neg', [self.c2], self.c1), - SpaceOperation('add', [self.c3, self.c3], self.c2), - SpaceOperation('immutable', [], self.c1), - SpaceOperation('immutable', [], self.c2), - SpaceOperation('immutable', [], self.c3)] + lst = [Annotation('add', [self.c1, self.c3], self.c2), + Annotation('neg', [self.c2], self.c1), + Annotation('add', [self.c3, self.c3], self.c2), + Annotation('immutable', [], self.c1), + Annotation('immutable', [], self.c2), + Annotation('immutable', [], self.c3)] a = AnnotationHeap(lst) # (c1) inter (c3) == (some new c4) - c, changeflag = a.merge(self.c1, self.c3) - self.assert_(changeflag) # because 'neg(..)=c1' is dropped + c = a.merge(self.c1, self.c3) self.failIfEqual(c, self.c1) self.failIfEqual(c, self.c3) - lst += [SpaceOperation('add', [c, self.c3], self.c2), - SpaceOperation('immutable', [], c)] + lst += [Annotation('add', [c, self.c3], self.c2), + Annotation('immutable', [], c)] self.assertSameSet(a, lst) +class TestTransaction(test.IntTestCase): + + def setUp(self): + self.c1 = XCell() + self.c2 = XCell() + self.c3 = XConstant(-2) + self.lst = [Annotation('add', [self.c1, self.c3], self.c2), + Annotation('neg', [self.c2], self.c1), + Annotation('add', [self.c3, self.c3], self.c2), + Annotation('immutable', [], self.c1), + Annotation('type', [self.c1], self.c3), + Annotation('type', [self.c3], self.c2)] + self.a = AnnotationHeap(self.lst) + + def test_get(self): + t = Transaction(self.a) + self.assertEquals(t.get('add', [self.c1, self.c3]), self.c2) + self.assertEquals(t.get('add', [self.c1, self.c2]), None) + self.assertEquals(t.get('sub', [self.c1, self.c3]), None) + + def test_get_type(self): + t = Transaction(self.a) + self.assertEquals(t.get_type(self.c1), -2) + self.assertEquals(t.get_type(self.c2), None) + self.assertEquals(t.get_type(self.c3), None) + + def test_set(self): + t = Transaction(self.a) + t.set('dummy', [self.c2], self.c1) + self.assertEquals(t.get('dummy', [self.c2]), self.c1) + + def test_set_type(self): + t = Transaction(self.a) + t.set_type(self.c2, int) + self.assertEquals(t.get('type', [self.c2]), XConstant(int)) + + def test_dep_set(self): + t = Transaction(self.a) + t.get('add', [self.c1, self.c3]) + t.get_type(self.c1) + t.set('dummy', [self.c2], self.c1) + new_ann = Annotation('dummy', [self.c2], self.c1) + self.cases = [] + for ann in self.a.enumerate(): + if ann == Annotation('add', [self.c1, self.c3], self.c2): + self.cases.append(0) + self.assertEquals(ann.forward_deps, [new_ann]) + elif ann == Annotation('type', [self.c1], self.c3): + self.cases.append(1) + self.assertEquals(ann.forward_deps, [new_ann]) + else: + self.assertEquals(ann.forward_deps, []) + self.cases.sort() + self.assertEquals(self.cases, [0, 1]) + + if __name__ == '__main__': test.main() Added: pypy/trunk/src/pypy/translator/test/test_annotation.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/translator/test/test_annotation.py Mon Nov 17 18:29:19 2003 @@ -0,0 +1,46 @@ + +import autopath +from pypy.tool import test + +from pypy.translator.annotation import XCell, XConstant, nothingyet, Annotation + + +class TestAnnotation(test.IntTestCase): + + def test_is_shared(self): + c1 = XCell() + c2 = XCell() + c3 = XCell() + c4 = XCell() + for a in (c1,c2,c3,c4): + for b in (c1,c2,c3,c4): + if a is not b: + self.failIfEqual(a, b) + c1.share(c2) + c4.share(c2) + c1.share(c3) + for a in (c1,c2,c3,c4): + for b in (c1,c2,c3,c4): + self.assert_(a.is_shared(b)) + self.assertEquals(a, b) + + def test_constant(self): + self.assertEquals(XConstant(5), XConstant(5)) + self.failIfEqual(XConstant(5), XConstant(6)) + self.failIfEqual(XConstant(5), XCell()) + + def test_annotation(self): + c1 = XCell() + c2 = XCell() + c3 = XCell() + a1 = Annotation('hello', [c1], c2) + a2 = Annotation('hello', [c1], c3) + + self.assertEquals(a1, a1) + self.failIfEqual (a1, a2) + c2.share(c3) + self.assertEquals(a1, a2) + + +if __name__ == '__main__': + test.main() From arigo at codespeak.net Mon Nov 17 20:26:32 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 17 Nov 2003 20:26:32 +0100 (MET) Subject: [pypy-svn] rev 2207 - pypy/trunk/src/pypy/objspace/flow Message-ID: <20031117192632.9CF2E5A439@thoth.codespeak.net> Author: arigo Date: Mon Nov 17 20:26:29 2003 New Revision: 2207 Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py pypy/trunk/src/pypy/objspace/flow/model.py Log: Finally fixed the variables in EggBlocks, which are no longer shared with other blocks. Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/flowcontext.py (original) +++ pypy/trunk/src/pypy/objspace/flow/flowcontext.py Mon Nov 17 20:26:29 2003 @@ -152,4 +152,16 @@ if w_result is not None: link = Link([w_result], self.graph.returnblock) self.crnt_block.closeblock(link) + self.fixeggblocks() + def fixeggblocks(self): + # EggBlocks reuse the variables of their previous block, + # which is deemed not acceptable for simplicity of the operations + # that will be performed later on the flow graph. + def fixegg(node): + if isinstance(node, EggBlock): + mapping = {} + for a in node.inputargs: + mapping[a] = Variable() + node.renamevariables(mapping) + traverse(fixegg, self.graph) Modified: pypy/trunk/src/pypy/objspace/flow/model.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/model.py (original) +++ pypy/trunk/src/pypy/objspace/flow/model.py Mon Nov 17 20:26:29 2003 @@ -40,6 +40,12 @@ result.append(op.result) return uniqueitems([w for w in result if isinstance(w, Variable)]) + def renamevariables(self, mapping): + self.inputargs[:] = [mapping.get(a, a) for a in self.inputargs] + for op in self.operations: + op.args[:] = [mapping.get(a, a) for a in op.args] + op.result = mapping.get(op.result, op.result) + def closeblock(self, *exits): assert self.exits == [], "block already closed" self.recloseblock(*exits) From arigo at codespeak.net Mon Nov 17 21:23:41 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 17 Nov 2003 21:23:41 +0100 (MET) Subject: [pypy-svn] rev 2208 - pypy/trunk/src/pypy/objspace/flow Message-ID: <20031117202341.E3A905A439@thoth.codespeak.net> Author: arigo Date: Mon Nov 17 21:23:41 2003 New Revision: 2208 Modified: pypy/trunk/src/pypy/objspace/flow/model.py Log: Oops, this was broken because EggBlocks actually shared their inputargs lists... Modified: pypy/trunk/src/pypy/objspace/flow/model.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/model.py (original) +++ pypy/trunk/src/pypy/objspace/flow/model.py Mon Nov 17 21:23:41 2003 @@ -16,6 +16,8 @@ def getargs(self): return self.startblock.inputargs + def getreturnvar(self): + return self.returnblock.inputargs[0] class Link: def __init__(self, args, target, exitcase=None): @@ -41,10 +43,12 @@ return uniqueitems([w for w in result if isinstance(w, Variable)]) def renamevariables(self, mapping): - self.inputargs[:] = [mapping.get(a, a) for a in self.inputargs] + self.inputargs = [mapping.get(a, a) for a in self.inputargs] for op in self.operations: - op.args[:] = [mapping.get(a, a) for a in op.args] + op.args = [mapping.get(a, a) for a in op.args] op.result = mapping.get(op.result, op.result) + for link in self.exits: + link.args = [mapping.get(a, a) for a in link.args] def closeblock(self, *exits): assert self.exits == [], "block already closed" From arigo at codespeak.net Mon Nov 17 22:00:09 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 17 Nov 2003 22:00:09 +0100 (MET) Subject: [pypy-svn] rev 2209 - in pypy/trunk/src/pypy/translator: . test Message-ID: <20031117210009.CC9FF5A439@thoth.codespeak.net> Author: arigo Date: Mon Nov 17 22:00:09 2003 New Revision: 2209 Modified: pypy/trunk/src/pypy/translator/annheap.py pypy/trunk/src/pypy/translator/annotation.py pypy/trunk/src/pypy/translator/annrpython.py pypy/trunk/src/pypy/translator/test/test_annrpython.py Log: Annotations seem to work. All tests pass again but I don't really understand why most of the translator.test.test_* don't run at all. Most of them are still broken. Interestingly only the few ones that are not are run :-) Modified: pypy/trunk/src/pypy/translator/annheap.py ============================================================================== --- pypy/trunk/src/pypy/translator/annheap.py (original) +++ pypy/trunk/src/pypy/translator/annheap.py Mon Nov 17 22:00:09 2003 @@ -87,19 +87,25 @@ self.annlist.append(rename(ann, newcell, resultcell)) return resultcell else: - # for mutable objects we must identify oldcell and newcell, - # and only keep the common annotations - newcell.share(oldcell) - # search again and list all annotations that talk about - # oldcell or newcell (same thing now) but are not in 'common' + if Annotation('immutable', [], oldcell) in self.annlist: + pass # old was immutable, don't touch it + elif Annotation('immutable', [], newcell) in self.annlist: + # new is immutable, old was not, inverse the roles + oldcell, newcell = newcell, oldcell + else: + # two mutable objects: we identify oldcell and newcell + newcell.share(oldcell) + # only keep the common annotations by listing all annotations + # to remove, which are the ones that talk about newcell but + # are not in 'common'. deleting = [] for ann in self.annlist: - if oldcell in ann.args or oldcell == ann.result: + if newcell in ann.args or newcell == ann.result: if ann not in common: deleting.append(ann) # apply changes self.simplify(kill=deleting) - return oldcell + return newcell def rename(ann, oldcell, newcell): @@ -153,6 +159,7 @@ def get_type(self, cell): """Get the type of 'cell', as specified by the annotations, or None.""" + # Returns None if cell is None. c = self.get('type', [cell]) if isinstance(c, XConstant): return c.value Modified: pypy/trunk/src/pypy/translator/annotation.py ============================================================================== --- pypy/trunk/src/pypy/translator/annotation.py (original) +++ pypy/trunk/src/pypy/translator/annotation.py Mon Nov 17 22:00:09 2003 @@ -112,6 +112,12 @@ return (isinstance(other, XConstant) and self.value == other.value or XCell.__eq__(self, other)) + def __repr__(self): + if self.shared: + return 'UNEXPECTEDLY SHARED %r' % XCell.__repr__(self) + else: + return 'XConstant %r' % self.value + # The more annotations about an XCell, the least general # it is. Extreme case: *all* possible annotations stand for an Modified: pypy/trunk/src/pypy/translator/annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/annrpython.py (original) +++ pypy/trunk/src/pypy/translator/annrpython.py Mon Nov 17 22:00:09 2003 @@ -1,132 +1,203 @@ from __future__ import generators -from pypy.translator.annset import AnnotationSet, Cell, deref +from pypy.translator.annheap import AnnotationHeap, Transaction +from pypy.translator.annotation import XCell, XConstant, Annotation from pypy.objspace.flow.model import Variable, Constant, SpaceOperation -#class GraphGlobalVariable(Variable): -# pass -class Annotator: - - def __init__(self, flowgraph): - self.flowgraph = flowgraph - - def build_types(self, input_arg_types): - input_ann = AnnotationSet() - for arg, arg_type in zip(self.flowgraph.getargs(), input_arg_types): - input_ann.set_type(arg, arg_type) - self.build_annotations(input_ann) - - def build_annotations(self,input_annotations): - self.annotated = {} - self.flowin(self.flowgraph.startblock,input_annotations) - - def get_return_value(self): - "Return the return_value variable." - return self.flowgraph.returnblock.inputargs[0] - - def get_variables_ann(self): - """Return a dict {Variable(): AnnotationSet()} mapping each variable - of the control flow graph to a set of annotations that apply to it.""" - # XXX this assumes that all variables are local to a single block, - # and returns for each variable the annotations for that block. - # This assumption is clearly false because of the EggBlocks. - # This has to be fixed anyway. - result = {} - for block, ann in self.annotated.items(): - for v in block.getvariables(): - #XXX assert v not in result, "Variables must not be shared" - result[v] = ann +class RPythonAnnotator: + """Block annotator for RPython. + See description in doc/transation/annotation.txt.""" + + def __init__(self): + self.heap = AnnotationHeap() + self.pendingblocks = [] # list of (block, list-of-XCells) + self.bindings = {} # map Variables/Constants to XCells/XConstants + self.annotated = {} # set of blocks already seen + + + #___ convenience high-level interface __________________ + + def build_types(self, flowgraph, input_arg_types): + """Recursively build annotations about the specific entry point.""" + # make input arguments and set their type + inputcells = [XCell() for arg in flowgraph.getargs()] + t = self.transaction() + for cell, arg_type in zip(inputcells, input_arg_types): + t.set_type(cell, arg_type) + # register the entry point + self.addpendingblock(flowgraph.startblock, inputcells) + # recursively proceed until no more pending block is left + self.complete() + + + #___ medium-level interface ____________________________ + + def addpendingblock(self, block, cells): + """Register an entry point into block with the given input cells.""" + self.pendingblocks.append((block, cells)) + + def transaction(self): + """Start a Transaction. Each new Annotation is marked as depending + on the Annotations queried for during the same Transation.""" + return Transaction(self.heap) + + def complete(self): + """Process pending blocks until none is left.""" + while self.pendingblocks: + # XXX don't know if it is better to pop from the head or the tail. + # let's do it breadth-first and pop from the head (oldest first). + # that's more stacklessy. + block, cells = self.pendingblocks.pop(0) + self.processblock(block, cells) + + def binding(self, arg): + "XCell or XConstant corresponding to the given Variable or Constant." + try: + return self.bindings[arg] + except KeyError: + if not isinstance(arg, Constant): + raise # propagate missing bindings for Variables + result = XConstant(arg.value) + self.consider_const(result, arg) + self.bindings[arg] = result + return result + + def bindnew(self, arg): + "Force the creation of a new binding for the given Variable." + assert isinstance(arg, Variable) + self.bindings[arg] = result = XCell() return result + def constant(self, value): + "Turn a value into an XConstant with the proper annotations." + return self.binding(Constant(value)) + + + #___ simplification (should be moved elsewhere?) _______ + + def reverse_binding(self, known_variables, cell): + """This is a hack.""" + # In simplify_calls, when we are trying to create the new + # SpaceOperation, all we have are XCells. But SpaceOperations take + # Variables, not XCells. Trouble is, we don't always have a Variable + # that just happens to be bound to the given XCells. A typical + # example would be if the tuple of arguments was created from another + # basic block or even another function. Well I guess there is no + # clean solution. + if isinstance(cell, XConstant): + return Constant(cell.value) + else: + for v in known_variables: + if self.bindings[v] == cell: + return v + else: + raise CannotSimplify + def simplify_calls(self): - for block, ann in self.annotated.iteritems(): + t = self.transaction() + for block in self.annotated: + known_variables = block.inputargs[:] newops = [] for op in block.operations: - if op.opname == "call": - w_func, w_varargs, w_kwargs = op.args - c = Cell() - ann.match(SpaceOperation('len', [w_varargs], c)) - if isinstance(c.content, Constant): - length = c.content.value - args_w = [w_func] + try: + if op.opname == "call": + func, varargs, kwargs = [self.binding(a) + for a in op.args] + c = t.get('len', [varargs]) + if not isinstance(c, XConstant): + raise CannotSimplify + length = c.value + v = self.reverse_binding(known_variables, func) + args = [v] for i in range(length): - c = Cell() - if not ann.match(SpaceOperation('getitem', [ - w_varargs, Constant(i)], c)): - break - args_w.append(deref(c)) - else: - op = SpaceOperation('simple_call', args_w, op.result) - # XXX check that w_kwargs is empty + c = t.get('getitem', [varargs, self.constant(i)]) + if c is None: + raise CannotSimplify + v = self.reverse_binding(known_variables, c) + args.append(v) + op = SpaceOperation('simple_call', args, op.result) + # XXX check that kwargs is empty + except CannotSimplify: + pass newops.append(op) + known_variables.append(op.result) block.operations = newops def simplify(self): self.simplify_calls() - #__________________________________________________ - def flowin(self, block, annotations): + #___ flowing annotations in blocks _____________________ + + def processblock(self, block, cells): if block not in self.annotated: - oldlen = None - self.annotated[block] = blockannotations = annotations + self.annotated[block] = True + self.flowin(block, cells) else: - blockannotations = self.annotated[block] - oldlen = len(blockannotations) - #import sys; print >> sys.stderr, block, blockannotations - #import sys; print >> sys.stderr, '/\\', annotations, '==>', - blockannotations.intersect(annotations) - #import sys; print >> sys.stderr, blockannotations - + # already seen; merge each of the block's input variable + newcells = [] + reflow = False + for a, cell2 in zip(block.inputargs, cells): + cell1 = self.bindings[a] # old binding + newcell = self.heap.merge(cell1, cell2) + newcells.append(newcell) + reflow = reflow or (newcell != cell1 and newcell != cell2) + # no need to re-flowin unless there is a completely new cell + if reflow: + self.flowin(block, newcells) + + def flowin(self, block, inputcells): + for a, cell in zip(block.inputargs, inputcells): + self.bindings[a] = cell for op in block.operations: - self.consider_op(op, blockannotations) - # assert monotonic decrease - assert (oldlen is None or len(blockannotations) <= oldlen), ( - block, oldlen, blockannotations) - if len(blockannotations) != oldlen: - for link in block.exits: - self.flownext(link,block) - - def consider_op(self,op,annotations): + self.consider_op(op) + for link in block.exits: + cells = [self.binding(a) for a in link.args] + self.addpendingblock(link.target, cells) + + + #___ creating the annotations based on operations ______ + + def consider_op(self,op): + argcells = [self.binding(a) for a in op.args] + resultcell = self.bindnew(op.result) consider_meth = getattr(self,'consider_op_'+op.opname,None) if consider_meth is not None: - consider_meth(op,annotations) + consider_meth(argcells, resultcell, self.transaction()) - def consider_op_add(self,op,annotations): - arg1,arg2 = op.args - type1 = annotations.get_type(arg1) - type2 = annotations.get_type(arg2) + def consider_op_add(self, (arg1,arg2), result, t): + type1 = t.get_type(arg1) + type2 = t.get_type(arg2) if type1 is int and type2 is int: - annotations.set_type(op.result, int) + t.set_type(result, int) elif type1 in (int, long) and type2 in (int, long): - annotations.set_type(op.result, long) + t.set_type(result, long) if type1 is str and type2 is str: - annotations.set_type(op.result, str) + t.set_type(result, str) if type1 is list and type2 is list: - annotations.set_type(op.result, list) + t.set_type(result, list) consider_op_inplace_add = consider_op_add - def consider_op_sub(self, op, annotations): - arg1, arg2 = op.args - type1 = annotations.get_type(arg1) - type2 = annotations.get_type(arg2) + def consider_op_sub(self, (arg1,arg2), result, t): + type1 = t.get_type(arg1) + type2 = t.get_type(arg2) if type1 is int and type2 is int: - annotations.set_type(op.result, int) + t.set_type(result, int) elif type1 in (int, long) and type2 in (int, long): - annotations.set_type(op.result, long) + t.set_type(result, long) consider_op_and_ = consider_op_sub # trailing underline consider_op_inplace_lshift = consider_op_sub - def consider_op_is_true(self, op, annotations): - annotations.set_type(op.result, bool) + def consider_op_is_true(self, (arg,), result, t): + t.set_type(result, bool) consider_op_not_ = consider_op_is_true - def consider_op_lt(self, op, annotations): - annotations.set_type(op.result, bool) + def consider_op_lt(self, (arg1,arg2), result, t): + t.set_type(result, bool) consider_op_le = consider_op_lt consider_op_eq = consider_op_lt @@ -134,74 +205,48 @@ consider_op_gt = consider_op_lt consider_op_ge = consider_op_lt - def consider_op_newtuple(self,op,annotations): - annotations.set_type(op.result,tuple) - ann = SpaceOperation("len",[op.result],Constant(len(op.args))) - annotations.add(ann) - for i in range(len(op.args)): - ann = SpaceOperation("getitem",[op.result,Constant(i)],op.args[i]) - annotations.add(ann) - - def consider_op_newlist(self, op, annotations): - annotations.set_type(op.result, list) - - def consider_op_newslice(self,op,annotations): - annotations.set_type(op.result, slice) - - def consider_op_getitem(self, op, annotations): - arg1,arg2 = op.args - type1 = annotations.get_type(arg1) - type2 = annotations.get_type(arg2) + def consider_op_newtuple(self, args, result, t): + t.set_type(result,tuple) + t.set("len", [result], self.constant(len(args))) + for i in range(len(args)): + t.set("getitem", [result, self.constant(i)], args[i]) + + def consider_op_newlist(self, args, result, t): + t.set_type(result, list) + + def consider_op_newslice(self, args, result, t): + t.set_type(result, slice) + + def consider_op_getitem(self, (arg1,arg2), result, t): + type1 = t.get_type(arg1) + type2 = t.get_type(arg2) if type1 in (list, tuple) and type2 is slice: - annotations.set_type(op.result, type1) + t.set_type(result, type1) - def consider_op_call(self, op, annotations): - func = op.args[0] - if not isinstance(func, Constant): + def consider_op_call(self, (func,varargs,kwargs), result, t): + if not isinstance(func, XConstant): return func = func.value # XXX: generalize this later if func is range: - annotations.set_type(op.result, list) + t.set_type(result, list) if func is pow: - varargs = op.args[1] - def getitem(var, i): - class NoMatch(Exception): pass - c = Cell() - match = annotations.match( - SpaceOperation('getitem', (var, Constant(i)), c)) - if match: return deref(c) - else: raise NoMatch - try: - tp1 = annotations.get_type(getitem(varargs, 0)) - tp2 = annotations.get_type(getitem(varargs, 1)) - if tp1 is int and tp2 is int: - annotations.set_type(op.result, int) - except NoMatch: - pass - - def consider_const(self,to_var,const,annotations): + tp1 = t.get_type(t.get('getitem', [varargs, self.constant(0)])) + tp2 = t.get_type(t.get('getitem', [varargs, self.constant(1)])) + if tp1 is int and tp2 is int: + t.set_type(result, int) + + def consider_const(self,to_var,const): + t = self.transaction() + t.set('immutable', [], to_var) if getattr(const, 'dummy', False): return # undefined local variables - annotations.set_type(to_var,type(const.value)) + t.set_type(to_var,type(const.value)) if isinstance(const.value, list): pass # XXX say something about the type of the elements elif isinstance(const.value, tuple): pass # XXX say something about the elements - def flownext(self,link,curblock): - renaming = {} - newannotations = AnnotationSet() - - for w_from,w_to in zip(link.args,link.target.inputargs): - if isinstance(w_from,Variable): - renaming.setdefault(w_from, []).append(w_to) - else: - self.consider_const(w_to,w_from,newannotations) - #import sys; print >> sys.stderr, self.annotated[curblock] - #import sys; print >> sys.stderr, renaming - for ann in self.annotated[curblock].enumerate(renaming): - newannotations.add(ann) - #import sys; print >> sys.stderr, newannotations - self.flowin(link.target,newannotations) +class CannotSimplify(Exception): + pass Modified: pypy/trunk/src/pypy/translator/test/test_annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/test_annrpython.py (original) +++ pypy/trunk/src/pypy/translator/test/test_annrpython.py Mon Nov 17 22:00:09 2003 @@ -3,15 +3,14 @@ from pypy.tool import test from pypy.tool.udir import udir -from pypy.translator.annrpython import Annotator +from pypy.translator.annrpython import RPythonAnnotator from pypy.objspace.flow.model import * class AnnonateTestCase(test.IntTestCase): def setUp(self): self.space = test.objspace('flow') - def make_ann(self, func): - """ make a pyrex-generated cfunction from the given func """ + def make_fun(self, func): import inspect try: func = func.im_func @@ -20,13 +19,12 @@ name = func.func_name funcgraph = self.space.build_flow(func) funcgraph.source = inspect.getsource(func) - return Annotator(funcgraph) + return funcgraph def reallyshow(self, graph): import os - from pypy.translator.test.make_dot import make_dot - from pypy.tool.udir import udir - dest = make_dot(graph, udir, 'ps') + from pypy.translator.tool.make_dot import make_dot + dest = make_dot('b', graph) os.system('gv %s' % str(dest)) def test_simple_func(self): @@ -42,11 +40,10 @@ fun = FunctionGraph("f", block) block.operations.append(op) block.closeblock(Link([result], fun.returnblock)) - a = Annotator(fun) - a.build_types([int]) - end_var = a.get_return_value() - end_ann = a.get_variables_ann()[end_var] - self.assertEquals(end_ann.get_type(end_var), int) + a = RPythonAnnotator() + a.build_types(fun, [int]) + end_cell = a.binding(fun.getreturnvar()) + self.assertEquals(a.transaction().get_type(end_cell), int) def test_while(self): """ @@ -70,12 +67,11 @@ Link([i], whileblock, True)) whileblock.operations.append(decop) whileblock.closeblock(Link([i], headerblock)) - - a = Annotator(fun) - a.build_types([int]) - end_var = a.get_return_value() - end_ann = a.get_variables_ann()[end_var] - self.assertEquals(end_ann.get_type(end_var), int) + + a = RPythonAnnotator() + a.build_types(fun, [int]) + end_cell = a.binding(fun.getreturnvar()) + self.assertEquals(a.transaction().get_type(end_cell), int) def test_while_sum(self): """ @@ -108,18 +104,17 @@ whileblock.operations.append(decop) whileblock.closeblock(Link([i, sum], headerblock)) - a = Annotator(fun) - #import sys; print >> sys.stderr, a.build_annotations(input_ann) - a.build_types([int]) - end_var = a.get_return_value() - end_ann = a.get_variables_ann()[end_var] - self.assertEquals(end_ann.get_type(end_var), int) + a = RPythonAnnotator() + a.build_types(fun, [int]) + end_cell = a.binding(fun.getreturnvar()) + self.assertEquals(a.transaction().get_type(end_cell), int) def test_simplify_calls(self): - a = self.make_ann(f_calls_g) - a.build_types([int]) + fun = self.make_fun(f_calls_g) + a = RPythonAnnotator() + a.build_types(fun, [int]) a.simplify_calls() - #self.reallyshow(a.flowgraph) + #self.reallyshow(fun) def g(n): return [0,1,2,n] From arigo at codespeak.net Tue Nov 18 00:04:36 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 18 Nov 2003 00:04:36 +0100 (MET) Subject: [pypy-svn] rev 2210 - pypy/trunk/src/pypy/translator Message-ID: <20031117230436.C7EFD5A770@thoth.codespeak.net> Author: arigo Date: Tue Nov 18 00:04:34 2003 New Revision: 2210 Modified: pypy/trunk/src/pypy/translator/annheap.py pypy/trunk/src/pypy/translator/annotation.py Log: Just in case, a few type-checking assertions. Modified: pypy/trunk/src/pypy/translator/annheap.py ============================================================================== --- pypy/trunk/src/pypy/translator/annheap.py (original) +++ pypy/trunk/src/pypy/translator/annheap.py Tue Nov 18 00:04:34 2003 @@ -160,6 +160,7 @@ def get_type(self, cell): """Get the type of 'cell', as specified by the annotations, or None.""" # Returns None if cell is None. + assert isinstance(cell, XCell) or cell is None c = self.get('type', [cell]) if isinstance(c, XConstant): return c.value Modified: pypy/trunk/src/pypy/translator/annotation.py ============================================================================== --- pypy/trunk/src/pypy/translator/annotation.py (original) +++ pypy/trunk/src/pypy/translator/annotation.py Tue Nov 18 00:04:34 2003 @@ -17,6 +17,10 @@ self.args = list(args) # list of XCells self.result = result # an XCell self.forward_deps = [] # annotations that depend on this one + # catch bugs involving confusion between Variables/Constants + # and XCells/XConstants + for cell in args + [result]: + assert isinstance(cell, XCell) def __eq__(self, other): return (self.__class__ is other.__class__ and From arigo at codespeak.net Tue Nov 18 00:05:15 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 18 Nov 2003 00:05:15 +0100 (MET) Subject: [pypy-svn] rev 2211 - pypy/trunk/src/pypy/translator Message-ID: <20031117230515.141C45A770@thoth.codespeak.net> Author: arigo Date: Tue Nov 18 00:05:14 2003 New Revision: 2211 Modified: pypy/trunk/src/pypy/translator/gencl.py pypy/trunk/src/pypy/translator/genpyrex.py pypy/trunk/src/pypy/translator/transform.py Log: Nice: making tests work without having to tweak the tests at all :-) Modified: pypy/trunk/src/pypy/translator/gencl.py ============================================================================== --- pypy/trunk/src/pypy/translator/gencl.py (original) +++ pypy/trunk/src/pypy/translator/gencl.py Tue Nov 18 00:05:14 2003 @@ -1,6 +1,6 @@ import autopath from pypy.objspace.flow.model import * -from pypy.translator.annrpython import Annotator +from pypy.translator.annrpython import RPythonAnnotator from pypy.translator.simplify import simplify_graph from pypy.translator.transform import transform_graph @@ -155,12 +155,17 @@ self.annotate(input_arg_types) transform_graph(self.ann) def annotate(self, input_arg_types): - ann = Annotator(self.fun) - ann.build_types(input_arg_types) + ann = RPythonAnnotator() + ann.build_types(self.fun, input_arg_types) ann.simplify() - self.ann = ann - def cur_annset(self): - return self.ann.annotated[self.cur_block] + self.setannotator(ann) + def setannotator(self, annotator): + self.ann = annotator + self.bindings = annotator.bindings + self.transaction = annotator.transaction() + def get_type(self, var): + cell = self.bindings.get(var) + return self.transaction.get_type(cell) def str(self, obj): if isinstance(obj, Variable): return obj.name @@ -215,9 +220,8 @@ for block in blocklist: tag = len(self.blockref) self.blockref[block] = tag - annset = self.ann.annotated[block] for var in block.getvariables(): - vardict[var] = annset.get_type(var) + vardict[var] = self.get_type(var) print "(", for var in vardict: if var in arglist: @@ -281,8 +285,7 @@ } def emit_typecase(self, table, *args): argreprs = tuple(map(self.str, args)) - annset = self.cur_annset() - argtypes = tuple(map(annset.get_type, args)) + argtypes = tuple(map(self.get_type, args)) if argtypes in table: trans = table[argtypes] print trans % argreprs Modified: pypy/trunk/src/pypy/translator/genpyrex.py ============================================================================== --- pypy/trunk/src/pypy/translator/genpyrex.py (original) +++ pypy/trunk/src/pypy/translator/genpyrex.py Tue Nov 18 00:05:14 2003 @@ -3,9 +3,9 @@ """ from pypy.interpreter.baseobjspace import ObjSpace -from pypy.objspace.flow.model import Variable, Constant, SpaceOperation +from pypy.objspace.flow.model import Variable, Constant from pypy.objspace.flow.model import mkentrymap -from pypy.translator.annrpython import Annotator +from pypy.translator.annrpython import RPythonAnnotator class Op: def __init__(self, operation, gen, block): @@ -143,16 +143,17 @@ oparity[opname] = arity self.ops = ops self.oparity = oparity - self.variables_ann = {} + self.bindings = {} def annotate(self, input_arg_types): - a = Annotator(self.functiongraph) - a.build_types(input_arg_types) + a = RPythonAnnotator() + a.build_types(self.functiongraph, input_arg_types) a.simplify() self.setannotator(a) def setannotator(self, annotator): - self.variables_ann = annotator.get_variables_ann() + self.bindings = annotator.bindings + self.transaction = annotator.transaction() def emitcode(self): self.blockids = {} @@ -184,13 +185,15 @@ #self.putline("# %r" % self.annotations) decllines = [] missing_decl = [] - for var in self.variables_ann: - if var not in fun.getargs(): - decl = self._vardecl(var) - if decl: - decllines.append(decl) - else: - missing_decl.append(self.get_varname(var)) + funargs = fun.getargs() + for block in self.blockids: + for var in block.getvariables(): + if var not in funargs: + decl = self._vardecl(var) + if decl: + decllines.append(decl) + else: + missing_decl.append(self.get_varname(var)) if missing_decl: missing_decl.sort() decllines.append('# untyped variables: ' + ' '.join(missing_decl)) @@ -201,11 +204,11 @@ self.lines.extend(functionbodylines) def get_type(self, var): - if var in self.variables_ann: - ann = self.variables_ann[var] - return ann.get_type(var) - elif isinstance(var, Constant): + if isinstance(var, Constant): return type(var.value) + elif var in self.bindings: + cell = self.bindings[var] + return self.transaction.get_type(cell) else: return None Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Tue Nov 18 00:05:14 2003 @@ -6,7 +6,6 @@ import autopath from pypy.objspace.flow.model import Variable, Constant, SpaceOperation -from pypy.translator.annrpython import Annotator # b = newlist(a) # d = mul(b, int c) @@ -14,7 +13,8 @@ def transform_allocate(self): """Transforms [a] * b to alloc_and_set(b, a) where b is int.""" - for block, ann in self.annotated.iteritems(): + t = self.transaction() + for block in self.annotated: operations = block.operations[:] n_op = len(operations) for i in range(0, n_op-1): @@ -24,7 +24,7 @@ len(op1.args) == 1 and op2.opname == 'mul' and op1.result is op2.args[0] and - ann.get_type(op2.args[1]) is int): + t.get_type(self.binding(op2.args[1])) is int): new_op = SpaceOperation('alloc_and_set', (op2.args[1], op1.args[0]), op2.result) @@ -36,14 +36,15 @@ def transform_slice(self): """Transforms a[b:c] to getslice(a, b, c).""" - for block, ann in self.annotated.iteritems(): + t = self.transaction() + for block in self.annotated: operations = block.operations[:] n_op = len(operations) for i in range(0, n_op-1): op1 = operations[i] op2 = operations[i+1] if (op1.opname == 'newslice' and - ann.get_type(op1.args[2]) is type(None) and + t.get_type(self.binding(op1.args[2])) is type(None) and op2.opname == 'getitem' and op1.result is op2.args[1]): new_op = SpaceOperation('getslice', From arigo at codespeak.net Tue Nov 18 00:16:03 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 18 Nov 2003 00:16:03 +0100 (MET) Subject: [pypy-svn] rev 2212 - pypy/trunk/src/pypy/translator Message-ID: <20031117231603.B88245A770@thoth.codespeak.net> Author: arigo Date: Tue Nov 18 00:16:02 2003 New Revision: 2212 Removed: pypy/trunk/src/pypy/translator/annset.py Modified: pypy/trunk/src/pypy/translator/translator.py Log: The new annotation scheme works reasonably well now. Note that RPythonAnnotator already supports whole-program annotating. The only missing piece is to actually detect the call operations, build (and cache) the new control flow graphs, and invoke the RPythonAnnotator.addpendingblock() method. D translator/annset.py this was the previous implementation of annotation sets. M translator/translator.py fixed a few imports. Deleted: /pypy/trunk/src/pypy/translator/annset.py ============================================================================== --- /pypy/trunk/src/pypy/translator/annset.py Tue Nov 18 00:16:02 2003 +++ (empty file) @@ -1,231 +0,0 @@ -from __future__ import generators -import weakref -from pypy.objspace.flow.model import Variable, Constant, SpaceOperation - - -class Cell: - """A logical variable. A Cell is an initially empty place that can - later contain a Variable or a Constant.""" - - # a Cell is "empty" if self.content is None, - # a Cell is "ground" otherwise. Once a Cell is ground its content - # cannot be changed any more. - - # Multiple empty Cells can be "shared"; a group of shared Cells act - # essentially like a single Cell in that setting one Cell content - # will give all Cells in the group the same content. - - def __init__(self): - self.content = None - self.shared = [] # list of weakrefs to Cells - # defining a group of shared cells - - def __repr__(self): - if self.content is None: - first = self.cellsingroup()[0] - return '' % (id(first),) - else: - return '' % (self.content,) - - def __eq__(self, other): - "Two sharing cells are identical." - if isinstance(other, Cell): - return self.is_shared(other) - else: - return self.content == other - - def __ne__(self, other): - return not (self == other) - - def cellsingroup(self): - if self.shared: - l = [s() for s in self.shared] - assert self in l - return [c for c in l if c is not None] - else: - return [self] - - def getsharelist(self): - if not self.shared: - self.shared = [weakref.ref(self)] - return self.shared - - def set(self, content): - if isinstance(content, Cell): - self.share(content) - elif self.content is None: - for c in self.cellsingroup(): - c.content = content - elif self.content != content: - raise ValueError, "cannot change the content of %r" % self - - def is_shared(self, other): - "Test if two cells are shared." - return self.shared is other.shared - - def share(self, other): - "Make two Cells share content." - if not self.is_shared(other): - if self.content is not None: - other.set(self.content) - elif other.content is not None: - self.set(other.content) - lst1 = self.getsharelist() - lst2 = other.getsharelist() - for s in lst2: - c = s() - if c is not None: - c.shared = lst1 - lst1.append(s) - - -class AnnotationSet: - - def __init__(self, annlist=[]): - self.byfunctor = {} - for ann in annlist: - self.add(ann) - - def copy(self): - a = AnnotationSet() - for functor, lst in self.byfunctor.items(): - a.byfunctor[functor] = lst[:] - return a - - def __repr__(self): - fulllist = list(self.enumerate()) - if fulllist: - lines = (['']) - else: - lines = ['' % (id(self),)] - return '\n'.join(lines) - - def __len__(self): - result = 0 - for lst in self.byfunctor.values(): - result += len(lst) - return result - - def match(self, pattern): - """Test if the annotation 'pattern' is present in the set. - This function sets all empty Cells of 'pattern', but it does not - change Cells in 'self'. All Cells of 'pattern' must be fresh.""" - functor = pattern.opname, len(pattern.args) - for ann in self.byfunctor.get(functor, []): - if same_functor_assign(pattern, ann): - return True - return False - - def intersect(self, otherset): - """Kill annotations in 'self' that are not present in 'otherset'. - It may set some Cells in 'self', but it does not change 'otherset'.""" - for annlist in self.byfunctor.values(): - for i in range(len(annlist)-1, -1, -1): - ann = annlist[i] - if not otherset.match(ann): - del annlist[i] - - def add(self, pattern): - """Add 'pattern' into 'self'. It may set some Cells in 'self' instead - of adding a new entry.""" - functor = pattern.opname, len(pattern.args) - annlist = self.byfunctor.setdefault(functor, []) - for ann in annlist: - if same_functor_assign(ann, pattern): - pattern = ann - break - else: - annlist.append(pattern) - - def enumerate(self, renaming=None): - """Yield a copy of all annotations in the set, possibly renaming - their variables according to a map {Variable: [list-of-Variables]}.""" - if renaming is None: - def renameall(list_w): - return [list_w] - else: - def rename(w): - if isinstance(w,Constant): - return [w] - else: - return renaming.get(w, []) - def renameall(list_w): - if list_w: - for w in rename(list_w[0]): - for tail_w in renameall(list_w[1:]): - yield [w] + tail_w - else: - yield [] - for lst in self.byfunctor.values(): - for ann in lst: - # we translate a single SpaceOperation(...) into either - # 0 or 1 or multiple ones, by replacing each variable - # used in the original operation by (in turn) any of - # the variables it can be renamed into - for list_w in renameall([ann.result] + ann.args): - result = list_w[0] - args = list_w[1:] - yield SpaceOperation(ann.opname,args,result) - - __iter__ = enumerate - - ### convenience methods ### - - def set_type(self, v, type): - self.add(SpaceOperation('type', [v], Constant(type))) - - def get_type(self, v): - if isinstance(v, Constant): - return type(v.value) - c = Cell() - self.match(SpaceOperation('type', [v], c)) - if isinstance(c.content, Constant): - return c.content.value - else: - return None - - def get_opresult(self, opname, args): - c = Cell() - self.match(SpaceOperation(opname, args, c)) - if isinstance(c.content, Constant): - return c.content.value - else: - return None - - -def annotation_assign(ann1, ann2): - """Assignment (ann1 = ann2). All empty cells in 'ann1' are set to the - value found in 'ann2'. Returns False if the two annotations are not - compatible.""" - functor1 = ann1.opname, len(ann1.args) - functor2 = ann2.opname, len(ann2.args) - return functor1 == functor2 and same_functor_assign(ann1, ann2) - -def same_functor_assign(ann1, ann2): - """Assignment (ann1 = ann2). All empty cells in 'ann1' are set to the - value found in 'ann2'. Returns False if the variables and constants - in the two annotations are not compatible. Assumes that the two - annotations have the same functor.""" - pairs = zip(ann1.args + [ann1.result], ann2.args + [ann2.result]) - for a1, a2 in pairs: - v1 = deref(a1) - if not isinstance(v1, Cell): - v2 = deref(a2) - if not isinstance(v2, Cell) and v2 != v1: - return False - # match! Set the Cells of ann1... - for a1, a2 in pairs: - v1 = deref(a1) - if isinstance(v1, Cell): - v1.set(a2) - return True - -def deref(x): - """If x is a Cell, return the content of the Cell, - or the Cell itself if empty. For other x, return x.""" - if isinstance(x, Cell) and x.content is not None: - return x.content - else: - return x Modified: pypy/trunk/src/pypy/translator/translator.py ============================================================================== --- pypy/trunk/src/pypy/translator/translator.py (original) +++ pypy/trunk/src/pypy/translator/translator.py Tue Nov 18 00:16:02 2003 @@ -32,8 +32,7 @@ import autopath from pypy.objspace.flow.model import * -from pypy.translator.annset import AnnotationSet, Cell -from pypy.translator.annotation import Annotator +from pypy.translator.annrpython import RPythonAnnotator from pypy.translator.simplify import simplify_graph from pypy.translator.genpyrex import GenPyrex from pypy.translator.gencl import GenCL @@ -73,8 +72,8 @@ Provides type information of arguments. Returns annotator. """ - self.annotator = Annotator(self.flowgraph) - self.annotator.build_types(input_args_types) + self.annotator = RPythonAnnotator() + self.annotator.build_types(self.flowgraph, input_args_types) return self.annotator def source(self): From arigo at codespeak.net Tue Nov 18 01:03:42 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 18 Nov 2003 01:03:42 +0100 (MET) Subject: [pypy-svn] rev 2213 - in pypy/trunk/src/pypy: objspace/flow translator Message-ID: <20031118000342.D01345A770@thoth.codespeak.net> Author: arigo Date: Tue Nov 18 01:03:41 2003 New Revision: 2213 Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py pypy/trunk/src/pypy/objspace/flow/model.py pypy/trunk/src/pypy/translator/annheap.py pypy/trunk/src/pypy/translator/annrpython.py pypy/trunk/src/pypy/translator/gencl.py pypy/trunk/src/pypy/translator/genpyrex.py Log: Replaced the hack for uninitialized local variables with its official equivalent, the 'nothingyet' XCell. This solves a few type annotation problems. For example, all variables of snippet.time_waster() with no exception are now successfully annotated as ints. Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/flowcontext.py (original) +++ pypy/trunk/src/pypy/objspace/flow/flowcontext.py Tue Nov 18 01:03:41 2003 @@ -70,8 +70,7 @@ self.w_globals = w_globals = space.wrap(globals) frame = code.create_frame(space, w_globals) formalargcount = code.getformalargcount() - dummy = Constant(None) - dummy.dummy = True + dummy = UndefinedConstant() arg_list = ([Variable() for i in range(formalargcount)] + [dummy] * (len(frame.fastlocals_w) - formalargcount)) frame.setfastscope(arg_list) Modified: pypy/trunk/src/pypy/objspace/flow/model.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/model.py (original) +++ pypy/trunk/src/pypy/objspace/flow/model.py Tue Nov 18 01:03:41 2003 @@ -75,7 +75,7 @@ self.value = value # a concrete value def __eq__(self, other): - return isinstance(other, Constant) and self.value == other.value + return self.__class__ is other.__class__ and self.value == other.value def __ne__(self, other): return not (self == other) @@ -86,6 +86,11 @@ def __repr__(self): return '%r' % (self.value,) +class UndefinedConstant(Constant): + # for local variables not defined yet. + def __init__(self): + Constant.__init__(self, None) + class SpaceOperation: def __init__(self, opname, args, result): self.opname = opname # operation name Modified: pypy/trunk/src/pypy/translator/annheap.py ============================================================================== --- pypy/trunk/src/pypy/translator/annheap.py (original) +++ pypy/trunk/src/pypy/translator/annheap.py Tue Nov 18 01:03:41 2003 @@ -1,4 +1,5 @@ from __future__ import generators +import types from annotation import Annotation, XCell, XConstant, nothingyet @@ -64,9 +65,10 @@ elif oldcell is nothingyet: return newcell else: + # any annotation or "constantness" about oldcell that must be killed? + deleting = isinstance(oldcell, XConstant) # find the annotations common to oldcell and newcell common = [] - deleting = [] # annotations about oldcell that must be killed for ann in self.annlist: if oldcell in ann.args or oldcell == ann.result: test1 = rename(ann, oldcell, newcell) @@ -74,7 +76,7 @@ if test1 in self.annlist and test2 in self.annlist: common.append(test1) else: - deleting.append(test1) + deleting = True # the involved objects are immutable if we have both # 'immutable() -> oldcell' and 'immutable() -> newcell' if Annotation('immutable', [], newcell) in common: @@ -170,7 +172,19 @@ def set_type(self, cell, type): """Register an annotation describing the type of the object 'cell'.""" self.set('type', [cell], XConstant(type)) + if type in immutable_types: + self.set('immutable', [], cell) def using(self, ann): """Mark 'ann' as used in this transaction.""" self.using_annotations.append(ann) + + +immutable_types = { + int: True, + long: True, + tuple: True, + str: True, + bool: True, + types.FunctionType: True, + } Modified: pypy/trunk/src/pypy/translator/annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/annrpython.py (original) +++ pypy/trunk/src/pypy/translator/annrpython.py Tue Nov 18 01:03:41 2003 @@ -1,8 +1,10 @@ from __future__ import generators from pypy.translator.annheap import AnnotationHeap, Transaction -from pypy.translator.annotation import XCell, XConstant, Annotation -from pypy.objspace.flow.model import Variable, Constant, SpaceOperation +from pypy.translator.annotation import XCell, XConstant, nothingyet +from pypy.translator.annotation import Annotation +from pypy.objspace.flow.model import Variable, Constant, UndefinedConstant +from pypy.objspace.flow.model import SpaceOperation class RPythonAnnotator: @@ -58,8 +60,11 @@ except KeyError: if not isinstance(arg, Constant): raise # propagate missing bindings for Variables - result = XConstant(arg.value) - self.consider_const(result, arg) + if isinstance(arg, UndefinedConstant): + result = nothingyet # undefined local variables + else: + result = XConstant(arg.value) + self.consider_const(result, arg) self.bindings[arg] = result return result @@ -131,23 +136,26 @@ #___ flowing annotations in blocks _____________________ def processblock(self, block, cells): + #print '* processblock', block, cells if block not in self.annotated: self.annotated[block] = True self.flowin(block, cells) else: # already seen; merge each of the block's input variable + oldcells = [] newcells = [] - reflow = False for a, cell2 in zip(block.inputargs, cells): cell1 = self.bindings[a] # old binding - newcell = self.heap.merge(cell1, cell2) - newcells.append(newcell) - reflow = reflow or (newcell != cell1 and newcell != cell2) - # no need to re-flowin unless there is a completely new cell - if reflow: + oldcells.append(cell1) + newcells.append(self.heap.merge(cell1, cell2)) + #print '** oldcells = ', oldcells + #print '** newcells = ', newcells + # re-flowin unless the newcells are equal to the oldcells + if newcells != oldcells: self.flowin(block, newcells) def flowin(self, block, inputcells): + #print '...' for a, cell in zip(block.inputargs, inputcells): self.bindings[a] = cell for op in block.operations: @@ -239,8 +247,6 @@ def consider_const(self,to_var,const): t = self.transaction() t.set('immutable', [], to_var) - if getattr(const, 'dummy', False): - return # undefined local variables t.set_type(to_var,type(const.value)) if isinstance(const.value, list): pass # XXX say something about the type of the elements Modified: pypy/trunk/src/pypy/translator/gencl.py ============================================================================== --- pypy/trunk/src/pypy/translator/gencl.py (original) +++ pypy/trunk/src/pypy/translator/gencl.py Tue Nov 18 01:03:41 2003 @@ -270,10 +270,9 @@ source = link.args target = link.target.inputargs print "(psetq", # parallel assignment - for item in zip(source, target): - init, var = map(self.str, item) - if var != init: - print var, init, + for s, t in zip(source, target): + if s != t and not isinstance(s, UndefinedConstant): + print self.str(t), self.str(s), print ")" self.emit_jump(link.target) typemap = { Modified: pypy/trunk/src/pypy/translator/genpyrex.py ============================================================================== --- pypy/trunk/src/pypy/translator/genpyrex.py (original) +++ pypy/trunk/src/pypy/translator/genpyrex.py Tue Nov 18 01:03:41 2003 @@ -3,7 +3,7 @@ """ from pypy.interpreter.baseobjspace import ObjSpace -from pypy.objspace.flow.model import Variable, Constant +from pypy.objspace.flow.model import Variable, Constant, UndefinedConstant from pypy.objspace.flow.model import mkentrymap from pypy.translator.annrpython import RPythonAnnotator @@ -292,16 +292,18 @@ def gen_link(self, prevblock, link): _str = self._str block = link.target - sourceargs = [_str(arg, prevblock) for arg in link.args] - targetargs = [_str(arg, block) for arg in block.inputargs] + sourceargs = link.args + targetargs = block.inputargs assert len(sourceargs) == len(targetargs) - # get rid of identity-assignments + # get rid of identity-assignments and assignments of UndefinedConstant sargs, targs = [], [] for s,t in zip(sourceargs, targetargs): - if s != t: + if s != t and not isinstance(s, UndefinedConstant): sargs.append(s) targs.append(t) if sargs: + sargs = [_str(arg, prevblock) for arg in sargs] + targs = [_str(arg, block) for arg in targs] self.putline("%s = %s" % (", ".join(targs), ", ".join(sargs))) self.gen_block(block) From sanxiyn at codespeak.net Tue Nov 18 04:56:55 2003 From: sanxiyn at codespeak.net (sanxiyn at codespeak.net) Date: Tue, 18 Nov 2003 04:56:55 +0100 (MET) Subject: [pypy-svn] rev 2214 - pypy/trunk/src/pypy/translator Message-ID: <20031118035655.D3C845A770@thoth.codespeak.net> Author: sanxiyn Date: Tue Nov 18 04:56:55 2003 New Revision: 2214 Modified: pypy/trunk/src/pypy/translator/transform.py Log: Corrected confusing comments that conflicts with docstring. Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Tue Nov 18 04:56:55 2003 @@ -5,11 +5,15 @@ """ import autopath +import types from pypy.objspace.flow.model import Variable, Constant, SpaceOperation -# b = newlist(a) -# d = mul(b, int c) -# --> d = alloc_and_set(c, a) +# [a] * b +# --> +# c = newlist(a) +# d = mul(c, int b) +# --> +# d = alloc_and_set(b, a) def transform_allocate(self): """Transforms [a] * b to alloc_and_set(b, a) where b is int.""" @@ -30,9 +34,12 @@ op2.result) block.operations[i:i+2] = [new_op] -# c = newslice(a, b, None) -# e = getitem(d, c) -# --> e = getslice(d, a, b) +# a[b:c] +# --> +# d = newslice(b, c, None) +# e = getitem(a, d) +# --> +# e = getslice(a, b, c) def transform_slice(self): """Transforms a[b:c] to getslice(a, b, c).""" @@ -44,7 +51,7 @@ op1 = operations[i] op2 = operations[i+1] if (op1.opname == 'newslice' and - t.get_type(self.binding(op1.args[2])) is type(None) and + t.get_type(self.binding(op1.args[2])) is types.NoneType and op2.opname == 'getitem' and op1.result is op2.args[1]): new_op = SpaceOperation('getslice', From sanxiyn at codespeak.net Tue Nov 18 05:26:14 2003 From: sanxiyn at codespeak.net (sanxiyn at codespeak.net) Date: Tue, 18 Nov 2003 05:26:14 +0100 (MET) Subject: [pypy-svn] rev 2215 - pypy/trunk/src/pypy/translator Message-ID: <20031118042614.E9F985A770@thoth.codespeak.net> Author: sanxiyn Date: Tue Nov 18 05:26:13 2003 New Revision: 2215 Modified: pypy/trunk/src/pypy/translator/annrpython.py pypy/trunk/src/pypy/translator/gencl.py pypy/trunk/src/pypy/translator/genpyrex.py pypy/trunk/src/pypy/translator/transform.py Log: simplify_calls is dead! long live transform.transform_simple_call! annrpython: simplify_calls removed transform: wrote transform_simple_call accordingly gencl, genpyrex: removed call to annotator.simplify(). Currently snippet.powerset is broken in gencl. Will fix shortly. Modified: pypy/trunk/src/pypy/translator/annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/annrpython.py (original) +++ pypy/trunk/src/pypy/translator/annrpython.py Tue Nov 18 05:26:13 2003 @@ -81,6 +81,10 @@ #___ simplification (should be moved elsewhere?) _______ + # it should be! + # now simplify_calls is moved to transform.py. + # i kept reverse_binding here for future(?) purposes though. --sanxiyn + def reverse_binding(self, known_variables, cell): """This is a hack.""" # In simplify_calls, when we are trying to create the new @@ -99,39 +103,6 @@ else: raise CannotSimplify - def simplify_calls(self): - t = self.transaction() - for block in self.annotated: - known_variables = block.inputargs[:] - newops = [] - for op in block.operations: - try: - if op.opname == "call": - func, varargs, kwargs = [self.binding(a) - for a in op.args] - c = t.get('len', [varargs]) - if not isinstance(c, XConstant): - raise CannotSimplify - length = c.value - v = self.reverse_binding(known_variables, func) - args = [v] - for i in range(length): - c = t.get('getitem', [varargs, self.constant(i)]) - if c is None: - raise CannotSimplify - v = self.reverse_binding(known_variables, c) - args.append(v) - op = SpaceOperation('simple_call', args, op.result) - # XXX check that kwargs is empty - except CannotSimplify: - pass - newops.append(op) - known_variables.append(op.result) - block.operations = newops - - def simplify(self): - self.simplify_calls() - #___ flowing annotations in blocks _____________________ Modified: pypy/trunk/src/pypy/translator/gencl.py ============================================================================== --- pypy/trunk/src/pypy/translator/gencl.py (original) +++ pypy/trunk/src/pypy/translator/gencl.py Tue Nov 18 05:26:13 2003 @@ -157,7 +157,6 @@ def annotate(self, input_arg_types): ann = RPythonAnnotator() ann.build_types(self.fun, input_arg_types) - ann.simplify() self.setannotator(ann) def setannotator(self, annotator): self.ann = annotator Modified: pypy/trunk/src/pypy/translator/genpyrex.py ============================================================================== --- pypy/trunk/src/pypy/translator/genpyrex.py (original) +++ pypy/trunk/src/pypy/translator/genpyrex.py Tue Nov 18 05:26:13 2003 @@ -148,7 +148,6 @@ def annotate(self, input_arg_types): a = RPythonAnnotator() a.build_types(self.functiongraph, input_arg_types) - a.simplify() self.setannotator(a) def setannotator(self, annotator): Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Tue Nov 18 05:26:13 2003 @@ -59,7 +59,42 @@ op2.result) block.operations[i:i+2] = [new_op] +# a(*b) +# --> +# c = newtuple(*b) +# d = newdict() +# e = call(function a, c, d) +# --> +# e = simple_call(a, *b) + +def transform_simple_call(self): + """Transforms a(*b) to simple_call(a, *b)""" + t = self.transaction() + for block in self.annotated: + operations = block.operations[:] + n_op = len(operations) + for i in range(0, n_op-2): + op1 = operations[i] + op2 = operations[i+1] + op3 = operations[i+2] + if not op3.args: continue + op3arg0type = t.get_type(self.binding(op3.args[0])) + if (op1.opname == 'newtuple' and + op2.opname == 'newdict' and + len(op2.args) == 0 and + op3.opname == 'call' and + op1.result is op3.args[1] and + op2.result is op3.args[2] and + # eek! + (op3arg0type is types.FunctionType or + op3arg0type is types.BuiltinFunctionType)): + new_op = SpaceOperation('simple_call', + (op3.args[0],) + tuple(op1.args), + op3.result) + block.operations[i:i+3] = [new_op] + def transform_graph(ann): """Apply set of transformations available.""" transform_allocate(ann) transform_slice(ann) + transform_simple_call(ann) From sanxiyn at codespeak.net Tue Nov 18 10:09:20 2003 From: sanxiyn at codespeak.net (sanxiyn at codespeak.net) Date: Tue, 18 Nov 2003 10:09:20 +0100 (MET) Subject: [pypy-svn] rev 2216 - pypy/trunk/src/pypy/translator Message-ID: <20031118090920.E3C7B5A770@thoth.codespeak.net> Author: sanxiyn Date: Tue Nov 18 10:09:20 2003 New Revision: 2216 Modified: pypy/trunk/src/pypy/translator/transform.py Log: small change couldn't fix powerset test in gencl easily... Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Tue Nov 18 10:09:20 2003 @@ -8,6 +8,8 @@ import types from pypy.objspace.flow.model import Variable, Constant, SpaceOperation +# XXX: Lots of duplicated codes. Fix this! + # [a] * b # --> # c = newlist(a) @@ -55,8 +57,8 @@ op2.opname == 'getitem' and op1.result is op2.args[1]): new_op = SpaceOperation('getslice', - (op2.args[0], op1.args[0], op1.args[1]), - op2.result) + (op2.args[0], op1.args[0], op1.args[1]), + op2.result) block.operations[i:i+2] = [new_op] # a(*b) From arigo at codespeak.net Tue Nov 18 12:33:31 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 18 Nov 2003 12:33:31 +0100 (MET) Subject: [pypy-svn] rev 2217 - pypy/trunk/src/pypy/translator/test Message-ID: <20031118113331.458565B726@thoth.codespeak.net> Author: arigo Date: Tue Nov 18 12:33:30 2003 New Revision: 2217 Removed: pypy/trunk/src/pypy/translator/test/test_annset.py Log: Obsolete. (Tests that can't be loaded seem to be silently ignored by the test suite, which is a very bad idea...) Deleted: /pypy/trunk/src/pypy/translator/test/test_annset.py ============================================================================== --- /pypy/trunk/src/pypy/translator/test/test_annset.py Tue Nov 18 12:33:30 2003 +++ (empty file) @@ -1,237 +0,0 @@ - -import autopath -from pypy.tool import test - -from pypy.translator.annset import Cell, AnnotationSet -from pypy.objspace.flow.model import Variable, Constant, SpaceOperation - - -class TestCell(test.IntTestCase): - - def test_set(self): - c1 = Cell() - v1 = Variable('v1') - c1.set(v1) - self.assertEquals(c1.content, v1) - c2 = Cell() - k2 = Constant(123) - c2.set(k2) - self.assertEquals(c2.content, k2) - self.assertRaises(ValueError, c1.set, k2) - self.assertRaises(ValueError, c2.set, v1) - self.assertRaises(ValueError, c1.set, c2) - self.assertRaises(ValueError, c2.set, c1) - - def test_share1(self): - k1 = Constant(-123) - c1 = Cell() - c2 = Cell() - c1.share(c2) - c2.set(k1) - self.assertEquals(c1.content, k1) - self.assertEquals(c2.content, k1) - - def test_share2(self): - k1 = Constant(-123) - c1 = Cell() - c2 = Cell() - c2.set(k1) - c1.share(c2) - self.assertEquals(c1.content, k1) - self.assertEquals(c2.content, k1) - - def test_share3(self): - k1 = Constant(-123) - c1 = Cell() - c2 = Cell() - c1.share(c2) - c1.set(k1) - self.assertEquals(c1.content, k1) - self.assertEquals(c2.content, k1) - - def test_share3(self): - k1 = Constant(-123) - c1 = Cell() - c2 = Cell() - c1.set(k1) - c1.share(c2) - self.assertEquals(c1.content, k1) - self.assertEquals(c2.content, k1) - - def test_is_shared(self): - c1 = Cell() - c2 = Cell() - c3 = Cell() - c4 = Cell() - for a in (c1,c2,c3,c4): - for b in (c1,c2,c3,c4): - if a is not b: - self.failIfEqual(a, b) - c1.share(c2) - c4.share(c2) - c1.share(c3) - for a in (c1,c2,c3,c4): - for b in (c1,c2,c3,c4): - self.assert_(a.is_shared(b)) - self.assertEquals(a, b) - - -class TestAnnotationSet(test.IntTestCase): - - def setUp(self): - self.v1 = Variable('v1') - self.v2 = Variable('v2') - self.v3 = Variable('v3') - self.k1 = Constant(102938) - self.k2 = Constant('foobar') - self.k3 = Constant(-2) - self.k4 = Constant(102938) - - def assertSameSet(self, a, b): - a = list(a) - b = list(b) - # try to reorder a to match b, without failing if the lists - # are different -- this will be checked by assertEquals() - for i in range(len(b)): - try: - j = i + a[i:].index(b[i]) - except ValueError: - pass - else: - a[i], a[j] = a[j], a[i] - self.assertEquals(a, b) - - def test_init(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v2], self.v3)] - a = AnnotationSet(lst) - self.assertSameSet(a, lst) - - def test_add(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v2], self.v3)] - a = AnnotationSet() - a.add(lst[1]) - a.add(lst[0]) - self.assertSameSet(a, lst) - a.add(lst[0]) - self.assertSameSet(a, lst) - a.add(lst[1]) - self.assertSameSet(a, lst) - - def test_add2(self): - c1 = Cell() - c2 = Cell() - c3 = Cell() - c4 = Cell() - c5 = Cell() - c6 = Cell() - c7 = Cell() - c8 = Cell() - c9 = Cell() - a = AnnotationSet() - op = SpaceOperation('add', [c1, c2], c3) - a.add(op) - self.assertSameSet(a, [op]) - op = SpaceOperation('add', [c4, self.k1], c5) - a.add(op) - self.assertSameSet(a, [op]) - op = SpaceOperation('add', [c6, self.k4], self.v3) - a.add(op) - self.assertSameSet(a, [op]) - op = SpaceOperation('add', [self.v1, self.k1], self.v3) - a.add(op) - self.assertSameSet(a, [op]) - - a.add(SpaceOperation('add', [self.v1, c7], self.v3)) - self.assertSameSet(a, [op]) - a.add(SpaceOperation('add', [self.v1, c9], c8)) - self.assertSameSet(a, [op]) - - def test_match1(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v1], self.k2), - SpaceOperation('neg', [self.v2], self.v3)] - a = AnnotationSet(lst) - for ann in lst: - self.assert_(a.match(ann)) - c = Cell() - self.assert_(a.match(SpaceOperation('add', [self.v1, self.k4], c))) - self.assertEquals(c.content, self.v2) - c = Cell() - c2 = Cell() - self.assert_(a.match(SpaceOperation('add', [self.v1, c], c2))) - self.assertEquals(c.content, self.k1) - self.assertEquals(c2.content, self.v2) - c = Cell() - self.assert_(a.match(SpaceOperation('neg', [c], self.v3))) - self.assertEquals(c.content, self.v2) - c = Cell() - self.failIf(a.match(SpaceOperation('add', [self.v2, self.k1], self.v2))) - self.failIf(a.match(SpaceOperation('add', [self.v2, self.k1], c))) - - def test_match2(self): - c1 = Cell() - c2 = Cell() - c3 = Cell() - c4 = Cell() - c4.share(c3) - c1.set(self.k4) - lst = [SpaceOperation('add', [self.v1, c1], self.v2), - SpaceOperation('neg', [self.v1], c2), - SpaceOperation('neg', [self.v2], c4)] - a = AnnotationSet(lst) - self.assert_(a.match(SpaceOperation('add', [self.v1, self.k1], self.v2))) - c = Cell() - self.assert_(a.match(SpaceOperation('add', [self.v1, self.k1], c))) - self.assertEquals(c.content, self.v2) - c = Cell() - self.assert_(a.match(SpaceOperation('neg', [self.v2], c))) - self.assert_(c.is_shared(c4)) - self.assert_(c.is_shared(c3)) - self.assert_(not c.is_shared(c2)) - self.assert_(not c.is_shared(c1)) - - self.assertEquals(c1.content, self.k1) - self.assertEquals(c2.content, None) - self.assertEquals(c3.content, None) - self.assertEquals(c4.content, None) - - def test_enumerate(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v1], self.k2), - SpaceOperation('neg', [self.v2], self.v3)] - a = AnnotationSet(lst) - self.assertSameSet(list(a.enumerate()), lst) - - def test_renaming(self): - lst = [SpaceOperation('add', [self.v1, self.k1], self.v2), - SpaceOperation('neg', [self.v1], self.k2), - SpaceOperation('neg', [self.v2], self.v3)] - renaming = {self.v1: [self.v3], - self.v2: [self.v2, self.v1]} - lst2 = [SpaceOperation('add', [self.v3, self.k1], self.v2), - SpaceOperation('add', [self.v3, self.k1], self.v1), - SpaceOperation('neg', [self.v3], self.k2)] - a = AnnotationSet(lst) - self.assertSameSet(list(a.enumerate(renaming)), lst2) - - def test_intersect(self): - lst = [SpaceOperation('type', [self.v2], self.k1), - SpaceOperation('type', [self.v1], self.k1), - SpaceOperation('type', [self.v3], self.k1), - ] - lst2 = [SpaceOperation('type', [self.v2], self.k4), - SpaceOperation('type', [self.v1], self.k4), - SpaceOperation('add', [self.v1, self.v2], self.v3), - ] - lst3 = [SpaceOperation('type', [self.v2], self.k1), - SpaceOperation('type', [self.v1], self.k1), - ] - a = AnnotationSet(lst) - a.intersect(AnnotationSet(lst2)) - self.assertSameSet(a, lst3) - - -if __name__ == '__main__': - test.main() From arigo at codespeak.net Tue Nov 18 13:07:21 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 18 Nov 2003 13:07:21 +0100 (MET) Subject: [pypy-svn] rev 2218 - in pypy/trunk/src/pypy/translator: . test Message-ID: <20031118120721.A87205B364@thoth.codespeak.net> Author: arigo Date: Tue Nov 18 13:07:21 2003 New Revision: 2218 Modified: pypy/trunk/src/pypy/translator/annheap.py pypy/trunk/src/pypy/translator/test/test_annheap.py Log: Transaction.get(opname, args): a None in args matches anything. Modified: pypy/trunk/src/pypy/translator/annheap.py ============================================================================== --- pypy/trunk/src/pypy/translator/annheap.py (original) +++ pypy/trunk/src/pypy/translator/annheap.py Tue Nov 18 13:07:21 2003 @@ -135,22 +135,33 @@ def get(self, opname, args): """Return the Cell with the annotation 'opname(args) -> Cell', - or None if there is no such annotation or several different ones.""" - matchann = None + or None if there is no such annotation or several different ones. + Hack to generalize: a None in the args matches anything.""" + # patch(arglist) -> arglist with None plugged where + # there is a None in the input 'args' + def patch(arglist): + return arglist + for i in range(len(args)): + if args[i] is None: + def patch(arglist, prevpatch=patch, i=i): + arglist = prevpatch(arglist)[:] + arglist[i] = None + return arglist + + matchann = [] for ann in self.heap.annlist: - if ann.opname == opname and ann.args == args: - if matchann is None: - matchann = ann # first possible annotation - elif matchann != ann: - return None # more than one annotation would match - if matchann is None: + if ann.opname == opname and patch(ann.args) == args: + matchann.append(ann) + if not matchann: return None else: - self.using(matchann) - return matchann.result - # a note about duplicate Annotations in annlist: their forward_deps - # lists will automatically be merged during the next simplify(), - # so that we only need to record the dependency from one of them. + result = matchann[0].result + for ann in matchann[1:]: + if result != ann.result: + return None # conflicting results + for ann in matchann: + self.using(ann) + return result def set(self, opname, args, result): """Put a new annotation into the AnnotationHeap.""" @@ -160,9 +171,11 @@ self.heap.annlist.append(ann) def get_type(self, cell): - """Get the type of 'cell', as specified by the annotations, or None.""" - # Returns None if cell is None. - assert isinstance(cell, XCell) or cell is None + """Get the type of 'cell', as specified by the annotations, or None. + Returns None if cell is None.""" + if cell is None: + return None + assert isinstance(cell, XCell) c = self.get('type', [cell]) if isinstance(c, XConstant): return c.value Modified: pypy/trunk/src/pypy/translator/test/test_annheap.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/test_annheap.py (original) +++ pypy/trunk/src/pypy/translator/test/test_annheap.py Tue Nov 18 13:07:21 2003 @@ -203,6 +203,13 @@ self.assertEquals(t.get('add', [self.c1, self.c2]), None) self.assertEquals(t.get('sub', [self.c1, self.c3]), None) + def test_get_None(self): + t = Transaction(self.a) + self.assertEquals(t.get('add', [self.c1, None]), self.c2) + self.assertEquals(t.get('add', [None, self.c3]), self.c2) + self.assertEquals(t.get('add', [self.c2, None]), None) + self.assertEquals(t.get('type', [None]), None) + def test_get_type(self): t = Transaction(self.a) self.assertEquals(t.get_type(self.c1), -2) From arigo at codespeak.net Tue Nov 18 14:01:51 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 18 Nov 2003 14:01:51 +0100 (MET) Subject: [pypy-svn] rev 2219 - in pypy/trunk/src/pypy/translator: . test Message-ID: <20031118130151.262905B364@thoth.codespeak.net> Author: arigo Date: Tue Nov 18 14:01:50 2003 New Revision: 2219 Modified: pypy/trunk/src/pypy/translator/annheap.py pypy/trunk/src/pypy/translator/annrpython.py pypy/trunk/src/pypy/translator/test/snippet.py pypy/trunk/src/pypy/translator/test/test_annrpython.py Log: Annotations about mutable objects work! See the new poor_man_rev_range() in snippet.py. The result is correctly inferred to be a list of integers. Hum, powerset fails in test_cltrans.py. Will investigate. Modified: pypy/trunk/src/pypy/translator/annheap.py ============================================================================== --- pypy/trunk/src/pypy/translator/annheap.py (original) +++ pypy/trunk/src/pypy/translator/annheap.py Tue Nov 18 14:01:50 2003 @@ -133,10 +133,7 @@ self.heap = heap self.using_annotations = [] # annotations that we have used - def get(self, opname, args): - """Return the Cell with the annotation 'opname(args) -> Cell', - or None if there is no such annotation or several different ones. - Hack to generalize: a None in the args matches anything.""" + def _list_annotations(self, opname, args): # patch(arglist) -> arglist with None plugged where # there is a None in the input 'args' def patch(arglist): @@ -152,6 +149,13 @@ for ann in self.heap.annlist: if ann.opname == opname and patch(ann.args) == args: matchann.append(ann) + return matchann + + def get(self, opname, args): + """Return the Cell with the annotation 'opname(args) -> Cell', + or None if there is no such annotation or several different ones. + Hack to generalize: a None in the args matches anything.""" + matchann = self._list_annotations(opname, args) if not matchann: return None else: @@ -163,6 +167,11 @@ self.using(ann) return result + def delete(self, opname, args): + """Kill the annotations 'opname(args) -> *'.""" + matchann = self._list_annotations(opname, args) + self.heap.simplify(kill=matchann) + def set(self, opname, args, result): """Put a new annotation into the AnnotationHeap.""" ann = Annotation(opname, args, result) Modified: pypy/trunk/src/pypy/translator/annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/annrpython.py (original) +++ pypy/trunk/src/pypy/translator/annrpython.py Tue Nov 18 14:01:50 2003 @@ -16,6 +16,12 @@ self.pendingblocks = [] # list of (block, list-of-XCells) self.bindings = {} # map Variables/Constants to XCells/XConstants self.annotated = {} # set of blocks already seen + # build default annotations + t = self.transaction() + self.any_immutable = XCell() + t.set('immutable', [], self.any_immutable) + self.any_int = XCell() + t.set_type(self.any_int, int) #___ convenience high-level interface __________________ @@ -156,8 +162,28 @@ t.set_type(result, str) if type1 is list and type2 is list: t.set_type(result, list) + # XXX propagate information about the type of the elements - consider_op_inplace_add = consider_op_add + def consider_op_inplace_add(self, (arg1,arg2), result, t): + type1 = t.get_type(arg1) + type2 = t.get_type(arg1) + if type1 is list and type2 is list: + # Annotations about the items of arg2 are merged with the ones about + # the items of arg1. arg2 is not modified during this operation. + # result is arg1. + result.share(arg1) + t.delete('len', [arg1]) + item1 = t.get('getitem', [arg1, None]) + if item1 is not None: + item2 = t.get('getitem', [arg2, None]) + if item2 is None: + item2 = XCell() # anything at all + item3 = self.heap.merge(item1, item2) + if item3 != item1: + t.delete('getitem', [arg1, None]) + t.set('getitem', [arg1, self.any_int], item3) + else: + self.consider_op_add((arg1,arg2), result, t) def consider_op_sub(self, (arg1,arg2), result, t): type1 = t.get_type(arg1) @@ -192,6 +218,11 @@ def consider_op_newlist(self, args, result, t): t.set_type(result, list) + t.set("len", [result], self.constant(len(args))) + item_cell = nothingyet + for a in args: + item_cell = self.heap.merge(item_cell, a) + t.set("getitem", [result, self.any_int], item_cell) def consider_op_newslice(self, args, result, t): t.set_type(result, slice) @@ -219,9 +250,7 @@ t = self.transaction() t.set('immutable', [], to_var) t.set_type(to_var,type(const.value)) - if isinstance(const.value, list): - pass # XXX say something about the type of the elements - elif isinstance(const.value, tuple): + if isinstance(const.value, tuple): pass # XXX say something about the elements Modified: pypy/trunk/src/pypy/translator/test/snippet.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/snippet.py (original) +++ pypy/trunk/src/pypy/translator/test/snippet.py Tue Nov 18 14:01:50 2003 @@ -105,6 +105,13 @@ lst.reverse() return lst +def poor_man_rev_range(i): + lst = [] + while i > 0: + i = i - 1 + lst += [i] + return lst + def simple_id(x): return x Modified: pypy/trunk/src/pypy/translator/test/test_annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/test_annrpython.py (original) +++ pypy/trunk/src/pypy/translator/test/test_annrpython.py Tue Nov 18 14:01:50 2003 @@ -6,6 +6,8 @@ from pypy.translator.annrpython import RPythonAnnotator from pypy.objspace.flow.model import * +from pypy.translator.test import snippet + class AnnonateTestCase(test.IntTestCase): def setUp(self): self.space = test.objspace('flow') @@ -109,12 +111,26 @@ end_cell = a.binding(fun.getreturnvar()) self.assertEquals(a.transaction().get_type(end_cell), int) - def test_simplify_calls(self): - fun = self.make_fun(f_calls_g) + #def test_simplify_calls(self): + # fun = self.make_fun(f_calls_g) + # a = RPythonAnnotator() + # a.build_types(fun, [int]) + # a.simplify_calls() + # #self.reallyshow(fun) + # XXX write test_transform.py + + def test_lists(self): + fun = self.make_fun(snippet.poor_man_rev_range) a = RPythonAnnotator() a.build_types(fun, [int]) - a.simplify_calls() - #self.reallyshow(fun) + # result should be a list of integers + t = a.transaction() + end_cell = a.binding(fun.getreturnvar()) + self.assertEquals(t.get_type(end_cell), list) + item_cell = t.get('getitem', [end_cell, None]) + self.failIf(item_cell is None) + self.assertEquals(t.get_type(item_cell), int) + def g(n): return [0,1,2,n] From sanxiyn at codespeak.net Tue Nov 18 16:50:20 2003 From: sanxiyn at codespeak.net (sanxiyn at codespeak.net) Date: Tue, 18 Nov 2003 16:50:20 +0100 (MET) Subject: [pypy-svn] rev 2220 - pypy/trunk/doc/sprintinfo Message-ID: <20031118155020.790A55B721@thoth.codespeak.net> Author: sanxiyn Date: Tue Nov 18 16:50:19 2003 New Revision: 2220 Modified: pypy/trunk/doc/sprintinfo/BerlinReport.txt Log: reST improvements Modified: pypy/trunk/doc/sprintinfo/BerlinReport.txt ============================================================================== --- pypy/trunk/doc/sprintinfo/BerlinReport.txt (original) +++ pypy/trunk/doc/sprintinfo/BerlinReport.txt Tue Nov 18 16:50:19 2003 @@ -3,15 +3,16 @@ Hi Florian, [Florian Schulze Sat, Oct 04, 2003 at 10:34:25PM +0200] -> Hi! -> -> How well did the sprint work out? -> -> I have seen that there is some pyrex code generation now and there are -> tests, but what where the results in this area during the sprint? -> -> Just a very short mail with some information would be grately -> appreciated. + + Hi! + + How well did the sprint work out? + + I have seen that there is some pyrex code generation now and there are + tests, but what where the results in this area during the sprint? + + Just a very short mail with some information would be grately + appreciated. Here is my take. Other mileages may vary so excuse me if i miss anything. @@ -82,20 +83,16 @@ To sum it up there are the following abstractions: +============ =============================================================== interpreter interpreting bytecode, dispatching operations on objects to - objectspace implementing operations on boxed objects - stdobjspace a concrete space implementing python's standard type system - flowobjspace a conrete space performing abstract/symbolic interpretation and producing a (bytecode-indepedent) flowmodel of execution - annotator analysing the flowmodel to infer types. - genpyrex taking the (annotated) flowmodel to generate pyrex-code - pyrex translating into an C-extension +============ =============================================================== As a consequence the former Ann(otation)Space has been ripped apart (partly into flowobjspace) and is gone now. Long live the flowspace. @@ -117,15 +114,16 @@ here is a (hopefully complete) list of people who attended and made all of the above possible: - Armin Rigo - Christian Tismer - Dinu Gherman - Guenter Jantzen - Jonathan David Riehl - Samuele Pedroni - Stephan Diehl - Tomek Meka - and shame on me if i forgot anyone (i am tired ...) +- Armin Rigo +- Christian Tismer +- Dinu Gherman +- Guenter Jantzen +- Jonathan David Riehl +- Samuele Pedroni +- Stephan Diehl +- Tomek Meka + +and shame on me if i forgot anyone (i am tired ...) And of course many many thanks to Laura Creighton (AB Strakt), Nicolas Chauvat (Logilab) and Alistair Burt (DFKI) who tried hard to From arigo at codespeak.net Tue Nov 18 18:47:23 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 18 Nov 2003 18:47:23 +0100 (MET) Subject: [pypy-svn] rev 2222 - pypy/trunk/src/pypy/translator Message-ID: <20031118174723.090F35B758@thoth.codespeak.net> Author: arigo Date: Tue Nov 18 18:47:23 2003 New Revision: 2222 Modified: pypy/trunk/src/pypy/translator/annrpython.py pypy/trunk/src/pypy/translator/transform.py Log: Fixed transform_simple_call(). Now test_cltrans.py passes all tests. Modified: pypy/trunk/src/pypy/translator/annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/annrpython.py (original) +++ pypy/trunk/src/pypy/translator/annrpython.py Tue Nov 18 18:47:23 2003 @@ -109,6 +109,11 @@ else: raise CannotSimplify + def simplify(self): + # Generic simpliciations + from pypy.translator import transform + transform.transform_simple_call(self) + #___ flowing annotations in blocks _____________________ @@ -227,6 +232,11 @@ def consider_op_newslice(self, args, result, t): t.set_type(result, slice) + def consider_op_newdict(self, args, result, t): + t.set_type(result, dict) + if not args: + t.set("len", [result], self.constant(0)) + def consider_op_getitem(self, (arg1,arg2), result, t): type1 = t.get_type(arg1) type2 = t.get_type(arg2) Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Tue Nov 18 18:47:23 2003 @@ -6,7 +6,9 @@ import autopath import types -from pypy.objspace.flow.model import Variable, Constant, SpaceOperation +from pypy.objspace.flow.model import SpaceOperation +from pypy.translator.annotation import XCell, XConstant +from pypy.translator.annrpython import CannotSimplify # XXX: Lots of duplicated codes. Fix this! @@ -70,30 +72,46 @@ # e = simple_call(a, *b) def transform_simple_call(self): - """Transforms a(*b) to simple_call(a, *b)""" + """Transforms call(a, (...), {}) to simple_call(a, ...)""" t = self.transaction() for block in self.annotated: - operations = block.operations[:] - n_op = len(operations) - for i in range(0, n_op-2): - op1 = operations[i] - op2 = operations[i+1] - op3 = operations[i+2] - if not op3.args: continue - op3arg0type = t.get_type(self.binding(op3.args[0])) - if (op1.opname == 'newtuple' and - op2.opname == 'newdict' and - len(op2.args) == 0 and - op3.opname == 'call' and - op1.result is op3.args[1] and - op2.result is op3.args[2] and - # eek! - (op3arg0type is types.FunctionType or - op3arg0type is types.BuiltinFunctionType)): - new_op = SpaceOperation('simple_call', - (op3.args[0],) + tuple(op1.args), - op3.result) - block.operations[i:i+3] = [new_op] + known_vars = block.inputargs[:] + operations = [] + for op in block.operations: + try: + if op.opname != 'call': + raise CannotSimplify + varargs_cell = self.binding(op.args[1]) + varkwds_cell = self.binding(op.args[2]) + + len_cell = t.get('len', [varargs_cell]) + if not isinstance(len_cell, XConstant): + raise CannotSimplify + nbargs = len_cell.value + arg_cells = [t.get('getitem', [varargs_cell, self.constant(j)]) + for j in range(nbargs)] + if None in arg_cells: + raise CannotSimplify + + len_cell = t.get('len', [varkwds_cell]) + if not isinstance(len_cell, XConstant): + raise CannotSimplify + nbkwds = len_cell.value + if nbkwds != 0: + raise CannotSimplify + + args = [self.reverse_binding(known_vars, c) for c in arg_cells] + args.insert(0, op.args[0]) + new_ops = [SpaceOperation('simple_call', args, op.result)] + + except CannotSimplify: + new_ops = [op] + + for op in new_ops: + operations.append(op) + known_vars.append(op.result) + + block.operations = operations def transform_graph(ann): """Apply set of transformations available.""" From arigo at codespeak.net Wed Nov 19 15:13:59 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 19 Nov 2003 15:13:59 +0100 (MET) Subject: [pypy-svn] rev 2223 - in pypy/trunk/src/pypy/objspace/flow: . test Message-ID: <20031119141359.52CDD5B839@thoth.codespeak.net> Author: arigo Date: Wed Nov 19 15:13:58 2003 New Revision: 2223 Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py pypy/trunk/src/pypy/objspace/flow/framestate.py pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py Log: Allows the flow objspace to create several blocks per joinpoint, duplicating some control paths of the source in several bits of the graph. This is used for try:except: and try:finally: handlers. In general it can be used whenever we don't want to merge too control flow paths too eagerly but prefer duplicating them. Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/flowcontext.py (original) +++ pypy/trunk/src/pypy/objspace/flow/flowcontext.py Wed Nov 19 15:13:58 2003 @@ -76,7 +76,7 @@ frame.setfastscope(arg_list) self.joinpoints = {} for joinpoint in code.getjoinpoints(): - self.joinpoints[joinpoint] = None + self.joinpoints[joinpoint] = [] # list of blocks initialblock = SpamBlock(FrameState(frame).copy()) self.pendingblocks = [initialblock] self.graph = FunctionGraph(code.co_name, initialblock) @@ -86,16 +86,21 @@ return next_instr = frame.next_instr if next_instr in self.joinpoints: - block = self.joinpoints[next_instr] currentstate = FrameState(frame) - if block is None: + # can 'currentstate' be merged with one of the blocks that + # already exist for this bytecode position? + for block in self.joinpoints[next_instr]: + newstate = block.framestate.union(currentstate) + if newstate is not None: + # yes + finished = newstate == block.framestate + break + else: + # no newstate = currentstate.copy() finished = False - else: - # there is already a block for this bytecode position, - # we merge its state with the new (current) state. - newstate = block.framestate.union(currentstate) - finished = newstate == block.framestate + block = None + if finished: newblock = block else: @@ -114,7 +119,7 @@ outputargs = block.framestate.getoutputargs(newstate) block.recloseblock(Link(outputargs, newblock)) newblock.patchframe(frame, self) - self.joinpoints[next_instr] = newblock + self.joinpoints[next_instr].insert(0, newblock) def guessbool(self, w_condition): if not isinstance(self.crnt_ops, ReplayList): Modified: pypy/trunk/src/pypy/objspace/flow/framestate.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/framestate.py (original) +++ pypy/trunk/src/pypy/objspace/flow/framestate.py Wed Nov 19 15:13:58 2003 @@ -1,4 +1,4 @@ -from pypy.interpreter.pyframe import PyFrame +from pypy.interpreter.pyframe import PyFrame, ControlFlowException from pypy.objspace.flow.model import * class FrameState: @@ -67,18 +67,11 @@ are also Variables in 'a', but 'a' may have more Variables. """ newstate = [] - for w1, w2 in zip(self.mergeable, other.mergeable): - if isinstance(w1, Variable): - w = Variable() # new fresh Variable - elif isinstance(w1, Constant): - if w1 == w2: - w = w1 - else: - w = Variable() # generalize different constants - else: - raise TypeError('union of %r and %r' % (w1.__class__.__name__, - w2.__class__.__name__)) - newstate.append(w) + try: + for w1, w2 in zip(self.mergeable, other.mergeable): + newstate.append(union(w1, w2)) + except UnionError: + return None return FrameState((newstate, self.nonmergeable)) def getoutputargs(self, targetstate): @@ -91,3 +84,27 @@ raise TypeError('output arg %r' % w_target.__class__.__name__) return result + +class UnionError(Exception): + "The two states should be merged." + +def union(w1, w2): + "Union of two variables or constants." + if isinstance(w1, Variable) or isinstance(w2, Variable): + return Variable() # new fresh Variable + if isinstance(w1, Constant) and isinstance(w2, Constant): + if w1 == w2: + return w1 + # ControlFlowException represent stack unrollers in the stack. + # They should not be merged because they will be unwrapped. + # This is needed for try:except: and try:finally:, though + # it makes the control flow a bit larger by duplicating the + # handlers. + dont_merge_w1 = isinstance(w1.value, ControlFlowException) + dont_merge_w2 = isinstance(w2.value, ControlFlowException) + if dont_merge_w1 or dont_merge_w2: + raise UnionError + else: + return Variable() # generalize different constants + raise TypeError('union of %r and %r' % (w1.__class__.__name__, + w2.__class__.__name__)) Modified: pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py Wed Nov 19 15:13:58 2003 @@ -139,5 +139,26 @@ x = self.codetest(self.nested_whiles) self.show(x) + #__________________________________________________________ + def break_continue(x): + result = [] + i = 0 + while 1: + i = i + 1 + try: + if i&1: + continue + if i >= x: + break + finally: + result.append(i) + i = i + 1 + return result + + def test_break_continue(self): + x = self.codetest(self.break_continue) + self.show(x) + + if __name__ == '__main__': test.main() From arigo at codespeak.net Wed Nov 19 17:46:43 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 19 Nov 2003 17:46:43 +0100 (MET) Subject: [pypy-svn] rev 2224 - pypy/trunk/src/pypy/translator/test Message-ID: <20031119164643.A527E5A2D9@thoth.codespeak.net> Author: arigo Date: Wed Nov 19 17:46:42 2003 New Revision: 2224 Modified: pypy/trunk/src/pypy/translator/test/snippet.py Log: Another example. (We should directly use all these examples in objspace.flow.test.test_objspace.) Modified: pypy/trunk/src/pypy/translator/test/snippet.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/snippet.py (original) +++ pypy/trunk/src/pypy/translator/test/snippet.py Wed Nov 19 17:46:42 2003 @@ -211,3 +211,43 @@ return 'yes' else: return 'no' + +def break_continue(x): + result = [] + i = 0 + while 1: + i = i + 1 + try: + if i&1: + continue + if i >= x: + break + finally: + result.append(i) + i = i + 1 + return result + +def reverse_3(lst): + try: + a, b, c = lst + except: + return 0, 0, 0 + return c, b, a + +def finallys(lst): + x = 1 + try: + x = 2 + try: + x = 3 + a, = lst + x = 4 + except KeyError: + return 5 + except ValueError: + return 6 + b, = lst + x = 7 + finally: + x = 8 + return x From arigo at codespeak.net Wed Nov 19 17:51:49 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 19 Nov 2003 17:51:49 +0100 (MET) Subject: [pypy-svn] rev 2225 - in pypy/trunk/src/pypy: interpreter objspace/flowobjspace/flow/test tool translator/tool Message-ID: <20031119165149.C83325A2D9@thoth.codespeak.net> Author: arigo Date: Wed Nov 19 17:51:48 2003 New Revision: 2225 Modified: pypy/trunk/src/pypy/interpreter/baseobjspace.py pypy/trunk/src/pypy/interpreter/pyframe.py pypy/trunk/src/pypy/objspace/flow/flowcontext.py pypy/trunk/src/pypy/objspace/flow/model.py pypy/trunk/src/pypy/objspace/flow/objspace.py pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py pypy/trunk/src/pypy/tool/test.py pypy/trunk/src/pypy/translator/tool/make_dot.py Log: Exception support for the flow objspace. M pypy/interpreter/pyframe.py M pypy/interpreter/baseobjspace.py M pypy/tool/test.py Added a flag 'full_exceptions' in object spaces. Allows FlowObjSpace to signal that it doesn't really support all exception operations like normalization and traceback printing. M pypy/objspace/flow/model.py M pypy/translator/tool/make_dot.py FunctionGraphs now have additional 'end' blocks, listed in the 'exceptblocks' dict, one per possible exception class that it can raise. M pypy/objspace/flow/test/test_objspace.py M pypy/objspace/flow/flowcontext.py M pypy/objspace/flow/objspace.py The necessary logic. The plan now is to have all operations tentatively raise exceptions like KeyError, IndexError, OverflowError, so that they appear in the CFG; a later simplification pass can apply the RPython specification that implicit uncaught exceptions should not occur. Modified: pypy/trunk/src/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/trunk/src/pypy/interpreter/baseobjspace.py (original) +++ pypy/trunk/src/pypy/interpreter/baseobjspace.py Wed Nov 19 17:51:48 2003 @@ -19,6 +19,8 @@ class ObjSpace: """Base class for the interpreter-level implementations of object spaces. http://codespeak.net/moin/pypy/moin.cgi/ObjectSpace""" + + full_exceptions = True # full support for exceptions (normalization & more) def __init__(self): "Basic initialization of objects." @@ -157,15 +159,15 @@ # Match identical items. if self.is_true(self.is_(w_exc_type, w_item)): return True - # Test within iterables (i.e. tuples) try: - exclst = self.unpackiterable(w_item) - check_list.extend(exclst) - except OperationError: - # w_item is not iterable; it should then be an Exception. # Match subclasses. if self.is_true(self.issubtype(w_exc_type, w_item)): return True + except OperationError: + # Assume that this is a TypeError: w_item not a type, + # and assume that w_item is then actually a tuple. + exclst = self.unpackiterable(w_item) + check_list.extend(exclst) return False def call_function(self, w_func, *args_w, **kw_w): Modified: pypy/trunk/src/pypy/interpreter/pyframe.py ============================================================================== --- pypy/trunk/src/pypy/interpreter/pyframe.py (original) +++ pypy/trunk/src/pypy/interpreter/pyframe.py Wed Nov 19 17:51:48 2003 @@ -159,10 +159,11 @@ # push the exception to the value stack for inspection by the # exception handler (the code after the except:) operationerr = unroller.args[0] - w_normalized = normalize_exception(frame.space, - operationerr.w_type, - operationerr.w_value) - w_type, w_value = frame.space.unpacktuple(w_normalized, 2) + w_type = operationerr.w_type + w_value = operationerr.w_value + if frame.space.full_exceptions: + w_normalized = normalize_exception(frame.space, w_type, w_value) + w_type, w_value = frame.space.unpacktuple(w_normalized, 2) # the stack setup is slightly different than in CPython: # instead of the traceback, we store the unroller object, # wrapped. Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/flowcontext.py (original) +++ pypy/trunk/src/pypy/objspace/flow/flowcontext.py Wed Nov 19 17:51:48 2003 @@ -2,6 +2,7 @@ from pypy.interpreter.miscutils import Stack from pypy.interpreter.pyframe \ import ControlFlowException, ExitFrame, PyFrame +from pypy.interpreter.error import OperationError from pypy.objspace.flow.model import * from pypy.objspace.flow.framestate import FrameState @@ -152,10 +153,16 @@ block.patchframe(frame, self) except ExitFrame: continue # restarting a dead SpamBlock - w_result = frame.eval(self) - if w_result is not None: - link = Link([w_result], self.graph.returnblock) + try: + w_result = frame.eval(self) + except OperationError, e: + exc_type = self.space.unwrap(e.w_type) # e.w_value ignored + link = Link([], self.graph.getexceptblock(exc_type)) self.crnt_block.closeblock(link) + else: + if w_result is not None: + link = Link([w_result], self.graph.returnblock) + self.crnt_block.closeblock(link) self.fixeggblocks() def fixeggblocks(self): Modified: pypy/trunk/src/pypy/objspace/flow/model.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/model.py (original) +++ pypy/trunk/src/pypy/objspace/flow/model.py Wed Nov 19 17:51:48 2003 @@ -13,12 +13,24 @@ self.returnblock = Block([return_var or Variable()]) self.returnblock.operations = () self.returnblock.exits = () + self.exceptblocks = {} # Blocks corresponding to exception results def getargs(self): return self.startblock.inputargs + def getreturnvar(self): return self.returnblock.inputargs[0] + def getexceptblock(self, exc_type): + try: + block = self.exceptblocks[exc_type] + except KeyError: + block = self.exceptblocks[exc_type] = Block([]) + block.exc_type = exc_type + block.operations = () + block.exits = () + return block + class Link: def __init__(self, args, target, exitcase=None): assert len(args) == len(target.inputargs), "output args mismatch" @@ -84,7 +96,7 @@ return hash(self.value) def __repr__(self): - return '%r' % (self.value,) + return '(%r)' % (self.value,) class UndefinedConstant(Constant): # for local variables not defined yet. Modified: pypy/trunk/src/pypy/objspace/flow/objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/objspace.py Wed Nov 19 17:51:48 2003 @@ -14,11 +14,17 @@ # ______________________________________________________________________ class FlowObjSpace(ObjSpace): + full_exceptions = False + def initialize(self): import __builtin__ self.w_builtins = Constant(__builtin__.__dict__) self.w_None = Constant(None) - self.w_KeyError = Constant(KeyError) + self.w_False = Constant(False) + self.w_True = Constant(True) + for exc in [KeyError, ValueError]: + clsname = exc.__name__ + setattr(self, 'w_'+clsname, Constant(exc)) #self.make_builtins() #self.make_sys() Modified: pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py Wed Nov 19 17:51:48 2003 @@ -21,9 +21,8 @@ def reallyshow(self, x): import os - from pypy.translator.test.make_dot import make_dot - from pypy.tool.udir import udir - dest = make_dot(x, udir, 'ps') + from pypy.translator.tool.make_dot import make_dot + dest = make_dot(x.name, x) os.system('gv %s' % str(dest)) def show(self, x): @@ -159,6 +158,50 @@ x = self.codetest(self.break_continue) self.show(x) + #__________________________________________________________ + def unpack_tuple(lst): + a, b, c = lst + + def test_unpack_tuple(self): + x = self.codetest(self.unpack_tuple) + self.show(x) + + #__________________________________________________________ + def reverse_3(lst): + try: + a, b, c = lst + except: + return 0, 0, 0 + else: + return c, b, a + + def test_reverse_3(self): + x = self.codetest(self.reverse_3) + self.show(x) + + #__________________________________________________________ + def finallys(lst): + x = 1 + try: + x = 2 + try: + x = 3 + a, = lst + x = 4 + except KeyError: + return 5 + except ValueError: + return 6 + b, = lst + x = 7 + finally: + x = 8 + return x + + def test_finallys(self): + x = self.codetest(self.finallys) + self.show(x) + if __name__ == '__main__': test.main() Modified: pypy/trunk/src/pypy/tool/test.py ============================================================================== --- pypy/trunk/src/pypy/tool/test.py (original) +++ pypy/trunk/src/pypy/tool/test.py Wed Nov 19 17:51:48 2003 @@ -44,7 +44,7 @@ def addError(self, test, err): # XXX not nice: from pypy.interpreter.baseobjspace import OperationError - if isinstance(err[1], OperationError): + if isinstance(err[1], OperationError) and test.space.full_exceptions: if err[1].match(test.space, test.space.w_AssertionError): self.addFailure(test, err) return @@ -58,7 +58,7 @@ def addError(self, test, err): from pypy.interpreter.baseobjspace import OperationError - if isinstance(err[1], OperationError): + if isinstance(err[1], OperationError) and test.space.full_exceptions: if err[1].match(test.space, test.space.w_AssertionError): self.addFailure(test, err) return @@ -121,7 +121,7 @@ self.stream.writeln(self.separator2) t1 = self._exc_info_to_string(err) t2 = '' - if isinstance(err[1], OperationError): + if isinstance(err[1], OperationError) and test.space.full_exceptions: t2 = '\nand at app-level:\n\n' sio = StringIO.StringIO() err[1].print_application_traceback(test.space, sio) Modified: pypy/trunk/src/pypy/translator/tool/make_dot.py ============================================================================== --- pypy/trunk/src/pypy/translator/tool/make_dot.py (original) +++ pypy/trunk/src/pypy/translator/tool/make_dot.py Wed Nov 19 17:51:48 2003 @@ -87,7 +87,7 @@ self.emit_node(name, label=data, shape="box", fillcolor="green", style="filled") #('%(name)s [fillcolor="green", shape=box, label="%(data)s"];' % locals()) self.emit_edge(name, self.blockname(funcgraph.startblock), 'startblock') - self.emit_edge(name, self.blockname(funcgraph.returnblock), 'returnblock', style="dashed") + #self.emit_edge(name, self.blockname(funcgraph.returnblock), 'returnblock', style="dashed") def visit_Block(self, block): # do the block itself @@ -100,6 +100,8 @@ if not numblocks: shape = "box" fillcolor="green" + if hasattr(block, 'exc_type'): + lines.insert(0, 'exc_type: %s' % block.exc_type.__name__) elif numblocks == 1: shape = "box" else: From arigo at codespeak.net Wed Nov 19 18:27:06 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 19 Nov 2003 18:27:06 +0100 (MET) Subject: [pypy-svn] rev 2226 - in pypy/trunk/src/pypy: objspace/flow objspace/flow/test translator translator/tool Message-ID: <20031119172706.4D3135B839@thoth.codespeak.net> Author: arigo Date: Wed Nov 19 18:27:05 2003 New Revision: 2226 Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py pypy/trunk/src/pypy/objspace/flow/objspace.py pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py pypy/trunk/src/pypy/translator/genpyrex.py pypy/trunk/src/pypy/translator/tool/stdoutcapture.py Log: Raising implicit exceptions. M pypy/objspace/flow/test/test_objspace.py M pypy/objspace/flow/flowcontext.py M pypy/objspace/flow/objspace.py Operations that can raise an exception generate two SpaceOperations and close the block: v3 = op(v1, v2...) v4 = exception(v3) exitswitch on v4 See description of the 'exception' operation in objspace.py. M pypy/translator/genpyrex.py Support for functions raising exceptions. M pypy/translator/tool/stdoutcapture.py Bug fix. Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/flowcontext.py (original) +++ pypy/trunk/src/pypy/objspace/flow/flowcontext.py Wed Nov 19 18:27:05 2003 @@ -122,22 +122,22 @@ newblock.patchframe(frame, self) self.joinpoints[next_instr].insert(0, newblock) - def guessbool(self, w_condition): + def guessbool(self, w_condition, cases=[False,True]): if not isinstance(self.crnt_ops, ReplayList): block = self.crnt_block vars = block.getvariables() - ifEgg = EggBlock(vars, block, True) - elseEgg = EggBlock(vars, block, False) - ifLink = Link(vars, ifEgg, True) - elseLink = Link(vars, elseEgg, False) + links = [] + for case in cases: + egg = EggBlock(vars, block, case) + self.pendingblocks.append(egg) + link = Link(vars, egg, case) + links.append(link) block.exitswitch = w_condition - block.closeblock(elseLink, ifLink) - # forked the graph. Note that elseLink comes before ifLink + block.closeblock(*links) + # forked the graph. Note that False comes before True by default # in the exits tuple so that (just in case we need it) we # actually have block.exits[False] = elseLink and # block.exits[True] = ifLink. - self.pendingblocks.append(ifEgg) - self.pendingblocks.append(elseEgg) raise ExitFrame(None) replaylist = self.crnt_ops assert replaylist.finished() Modified: pypy/trunk/src/pypy/objspace/flow/objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/objspace.py Wed Nov 19 18:27:05 2003 @@ -106,6 +106,10 @@ # ______________________________________________________________________ +implicit_exceptions = { + 'getitem': [IndexError], + } + def make_op(name, symbol, arity, specialnames): if hasattr(FlowObjSpace, name): return # Shouldn't do it @@ -121,6 +125,8 @@ else: if debug: print >> sys.stderr, "XXX missing operator:", name + exceptions = implicit_exceptions.get(name) + def generic_operator(self, *args_w): assert len(args_w) == arity, name+" got the wrong number of arguments" args = [] @@ -143,7 +149,19 @@ return self.wrap(result) #print >> sys.stderr, 'Variable operation', name, args_w - return self.do_operation(name, *args_w) + w_result = self.do_operation(name, *args_w) + if exceptions: + # the 'exception(w_result)' operation is a bit strange, it is + # meant to check if w_result is a correct result or if its + # computation actually resulted in an exception. For now this + # is an approximation of checking if w_result is NULL, and + # using PyErr_Occurred() to get the current exception if so. + w_curexc = self.do_operation('exception', w_result) + context = self.getexecutioncontext() + outcome = context.guessbool(w_curexc, [None] + exceptions) + if outcome is not None: + raise OperationError(self.wrap(outcome), self.w_None) + return w_result setattr(FlowObjSpace, name, generic_operator) Modified: pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py Wed Nov 19 18:27:05 2003 @@ -202,6 +202,18 @@ x = self.codetest(self.finallys) self.show(x) + #__________________________________________________________ + def implicitIndexError(lst): + try: + x = lst[5] + except IndexError: + return 'catch' + return lst[3] # not caught + + def test_implicitIndexError(self): + x = self.codetest(self.implicitIndexError) + self.reallyshow(x) + if __name__ == '__main__': test.main() Modified: pypy/trunk/src/pypy/translator/genpyrex.py ============================================================================== --- pypy/trunk/src/pypy/translator/genpyrex.py (original) +++ pypy/trunk/src/pypy/translator/genpyrex.py Wed Nov 19 18:27:05 2003 @@ -133,6 +133,12 @@ def op_is_true(self): return "%s = not not %s" % (self.resultname, self.argnames[0]) + def op_exception(self): + # Cheat! This cannot really detect an exception because any + # exception would already have been raised by Pyrex in the previous + # instructions. + return "%s = None #exception(%s)" % (self.resultname, self.argnames[0]) + class GenPyrex: def __init__(self, functiongraph): self.functiongraph = functiongraph @@ -285,6 +291,8 @@ self.indent += 1 self.gen_link(block, exit) self.indent -= 1 + elif hasattr(block, 'exc_type'): + self.putline("raise %s" % block.exc_type.__name__) else: self.putline("return %s" % self._str(block.inputargs[0], block)) Modified: pypy/trunk/src/pypy/translator/tool/stdoutcapture.py ============================================================================== --- pypy/trunk/src/pypy/translator/tool/stdoutcapture.py (original) +++ pypy/trunk/src/pypy/translator/tool/stdoutcapture.py Wed Nov 19 18:27:05 2003 @@ -19,12 +19,12 @@ # make new stdout/stderr files if needed self.localoutfd = os.dup(1) self.localerrfd = os.dup(2) - if sys.stdout.fileno() == 1: + if hasattr(sys.stdout, 'fileno') and sys.stdout.fileno() == 1: self.saved_stdout = sys.stdout sys.stdout = os.fdopen(self.localoutfd, 'w', 1) else: self.saved_stdout = None - if sys.stderr.fileno() == 2: + if hasattr(sys.stderr, 'fileno') and sys.stderr.fileno() == 2: self.saved_stderr = sys.stderr sys.stderr = os.fdopen(self.localerrfd, 'w', 0) else: From arigo at codespeak.net Wed Nov 19 18:38:23 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 19 Nov 2003 18:38:23 +0100 (MET) Subject: [pypy-svn] rev 2227 - in pypy/trunk/src/pypy: objspace/flow translator Message-ID: <20031119173823.E8E4B5B839@thoth.codespeak.net> Author: arigo Date: Wed Nov 19 18:38:23 2003 New Revision: 2227 Modified: pypy/trunk/src/pypy/objspace/flow/objspace.py pypy/trunk/src/pypy/translator/genpyrex.py Log: Yup! Finally got rid of the awful 'next_and_flag' operation. Now it is, expectedly, just a 'next' that can raise a StopIteration. Modified: pypy/trunk/src/pypy/objspace/flow/objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/objspace.py Wed Nov 19 18:38:23 2003 @@ -96,13 +96,14 @@ return context.guessbool(w_truthvalue) def next(self, w_iter): - w_tuple = self.do_operation("next_and_flag", w_iter) - w_flag = self.do_operation("getitem", w_tuple, Constant(1)) + w_item = self.do_operation("next", w_iter) + w_curexc = self.do_operation('exception', w_item) context = self.getexecutioncontext() - if context.guessbool(w_flag): - return self.do_operation("getitem", w_tuple, Constant(0)) - else: + outcome = context.guessbool(w_curexc, [None, StopIteration]) + if outcome is StopIteration: raise NoValue + else: + return w_item # ______________________________________________________________________ Modified: pypy/trunk/src/pypy/translator/genpyrex.py ============================================================================== --- pypy/trunk/src/pypy/translator/genpyrex.py (original) +++ pypy/trunk/src/pypy/translator/genpyrex.py Wed Nov 19 18:38:23 2003 @@ -50,15 +50,15 @@ return "%s = %s(%s)" % (self.resultname, self.op.opname, ", ".join(self.argnames)) - def op_next_and_flag(self): + def op_next(self): lines = [] args = self.argnames lines.append("try:") - lines.append(" _nextval = %s.next()" % args[0]) + lines.append(" %s = %s.next()" % (self.resultname, args[0])) lines.append("except StopIteration:") - lines.append(" %s = None, 0" % self.resultname) + lines.append(" last_exc = StopIteration") lines.append("else:") - lines.append(" %s = _nextval, 1" % self.resultname) + lines.append(" last_exc = None") return "\n".join(lines) def op_getitem(self): @@ -134,10 +134,7 @@ return "%s = not not %s" % (self.resultname, self.argnames[0]) def op_exception(self): - # Cheat! This cannot really detect an exception because any - # exception would already have been raised by Pyrex in the previous - # instructions. - return "%s = None #exception(%s)" % (self.resultname, self.argnames[0]) + return "%s, last_exc = last_exc, None" % (self.resultname,) class GenPyrex: def __init__(self, functiongraph): @@ -178,6 +175,7 @@ currentlines = self.lines self.lines = [] self.indent += 1 + self.putline("last_exc = None") self.gen_block(fun.startblock) self.indent -= 1 # emit the header after the body From arigo at codespeak.net Wed Nov 19 19:10:18 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 19 Nov 2003 19:10:18 +0100 (MET) Subject: [pypy-svn] rev 2228 - pypy/trunk/src/pypy/translator Message-ID: <20031119181018.499F95B8F8@thoth.codespeak.net> Author: arigo Date: Wed Nov 19 19:10:11 2003 New Revision: 2228 Modified: pypy/trunk/src/pypy/translator/gencl.py pypy/trunk/src/pypy/translator/transform.py Log: Fixed gencl.py for the new exception support. Also changed a detail in transform.py: "probably-dead" operations are no longer removed because it breaks the following (psetq) which see undefined variables for the results of removed operations. This should be solved in a better way. Modified: pypy/trunk/src/pypy/translator/gencl.py ============================================================================== --- pypy/trunk/src/pypy/translator/gencl.py (original) +++ pypy/trunk/src/pypy/translator/gencl.py Wed Nov 19 19:10:11 2003 @@ -118,10 +118,15 @@ s = self.str result, (seq,) = self.result, self.args print "(setq", s(result), "(make-iterator", s(seq), "))" - def op_next_and_flag(self): + def op_next(self): s = self.str result, (iterator,) = self.result, self.args - print "(setq", s(result), "(funcall", s(iterator), "))" + print "(let ((result (funcall", s(iterator), ")))" + print " (setq", s(result), "(car result))" + print " (setq last-exc (cdr result)))" + def op_exception(self): + s = self.str + print "(psetq", s(self.result), "last-exc last-exc nil)" builtin_map = { pow: "expt", range: "python-range", @@ -171,7 +176,7 @@ elif isinstance(obj, Constant): return self.conv(obj.value) else: - return "#<" + return "#<%r>" % (obj,) def conv(self, val): if isinstance(val, bool): # should precedes int if val: @@ -187,8 +192,10 @@ val.replace("\"", "\\\"") val = '"' + val + '"' return val + elif isinstance(val, type(Exception)) and issubclass(val, Exception): + return "'%s" % val.__name__ else: - return "#<" + return "#<%r>" % (val,) def emitcode(self): import sys from cStringIO import StringIO @@ -221,7 +228,7 @@ self.blockref[block] = tag for var in block.getvariables(): vardict[var] = self.get_type(var) - print "(", + print "( last-exc", for var in vardict: if var in arglist: print "(", self.str(var), self.str(var), ")", @@ -233,6 +240,7 @@ tp = vardict[var] if tp: print ";;", self.str(var), "is", tp.__name__ + print "(setq last-exc nil)" for block in blocklist: self.emit_block(block) print ")" @@ -249,16 +257,32 @@ self.emit_link(exits[0]) elif len(exits) > 1: # only works in the current special case - assert len(exits) == 2 - assert exits[0].exitcase == False - assert exits[1].exitcase == True - print "(if", self.str(block.exitswitch) - print "(progn" - self.emit_link(exits[1]) - print ") ; else" - print "(progn" - self.emit_link(exits[0]) - print "))" + if (len(exits) == 2 and + exits[0].exitcase == False and + exits[1].exitcase == True): + print "(if", self.str(block.exitswitch) + print "(progn" + self.emit_link(exits[1]) + print ") ; else" + print "(progn" + self.emit_link(exits[0]) + print "))" + else: + # this is for the more general case. The previous special case + # shouldn't be needed but in Python 2.2 we can't tell apart + # 0 vs nil and 1 vs t :-( + for exit in exits[:-1]: + print "(if (equalp", self.str(block.exitswitch), + print self.conv(exit.exitcase), ')' + print "(progn" + self.emit_link(exit) + print ")" + print "(progn ; else should be", self.conv(exits[-1].exitcase) + self.emit_link(exits[-1]) + print ")" * len(exits) + elif hasattr(block, 'exc_type'): + excname = block.exc_type.__name__ + print "(something-like-throw-exception '%s)" % excname else: retval = self.str(block.inputargs[0]) print "(return", retval, ")" @@ -309,8 +333,8 @@ (let ((i 0)) (lambda () (if (< i (length seq)) - (let ((v (elt seq i))) (incf i) (list v t)) - (list nil nil))))) + (let ((v (elt seq i))) (incf i) (cons v nil)) + (cons nil 'StopIteration))))) (defun python-slice (seq start end) (let ((l (length seq))) (if (not start) (setf start 0)) Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Wed Nov 19 19:10:11 2003 @@ -36,7 +36,7 @@ new_op = SpaceOperation('alloc_and_set', (op2.args[1], op1.args[0]), op2.result) - block.operations[i:i+2] = [new_op] + block.operations[i+1:i+2] = [new_op] # a[b:c] # --> @@ -61,7 +61,7 @@ new_op = SpaceOperation('getslice', (op2.args[0], op1.args[0], op1.args[1]), op2.result) - block.operations[i:i+2] = [new_op] + block.operations[i+1:i+2] = [new_op] # a(*b) # --> From arigo at codespeak.net Wed Nov 19 19:11:59 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 19 Nov 2003 19:11:59 +0100 (MET) Subject: [pypy-svn] rev 2229 - pypy/trunk/src/pypy/objspace/flow/test Message-ID: <20031119181159.E75225B8F8@thoth.codespeak.net> Author: arigo Date: Wed Nov 19 19:11:47 2003 New Revision: 2229 Modified: pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py Log: Ok test, we've seen you, you're nice, now don't show up any more. Modified: pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/test/test_objspace.py Wed Nov 19 19:11:47 2003 @@ -212,7 +212,7 @@ def test_implicitIndexError(self): x = self.codetest(self.implicitIndexError) - self.reallyshow(x) + self.show(x) if __name__ == '__main__': From arigo at codespeak.net Wed Nov 19 19:39:09 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 19 Nov 2003 19:39:09 +0100 (MET) Subject: [pypy-svn] rev 2230 - in pypy/trunk/src/pypy: objspace/flow translator Message-ID: <20031119183909.123E15B8F8@thoth.codespeak.net> Author: arigo Date: Wed Nov 19 19:39:05 2003 New Revision: 2230 Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py pypy/trunk/src/pypy/objspace/flow/model.py pypy/trunk/src/pypy/objspace/flow/objspace.py pypy/trunk/src/pypy/translator/gencl.py pypy/trunk/src/pypy/translator/simplify.py Log: The 'value' associated with the exception is now preserved. Implicit exceptions use a dummy value, which -- when uncaught -- is detected in simplify.py by the new remove_implicit_exceptions(). Modified: pypy/trunk/src/pypy/objspace/flow/flowcontext.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/flowcontext.py (original) +++ pypy/trunk/src/pypy/objspace/flow/flowcontext.py Wed Nov 19 19:39:05 2003 @@ -156,8 +156,8 @@ try: w_result = frame.eval(self) except OperationError, e: - exc_type = self.space.unwrap(e.w_type) # e.w_value ignored - link = Link([], self.graph.getexceptblock(exc_type)) + exc_type = self.space.unwrap(e.w_type) + link = Link([e.w_value], self.graph.getexceptblock(exc_type)) self.crnt_block.closeblock(link) else: if w_result is not None: Modified: pypy/trunk/src/pypy/objspace/flow/model.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/model.py (original) +++ pypy/trunk/src/pypy/objspace/flow/model.py Wed Nov 19 19:39:05 2003 @@ -25,7 +25,7 @@ try: block = self.exceptblocks[exc_type] except KeyError: - block = self.exceptblocks[exc_type] = Block([]) + block = self.exceptblocks[exc_type] = Block([Variable()]) block.exc_type = exc_type block.operations = () block.exits = () Modified: pypy/trunk/src/pypy/objspace/flow/objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/objspace.py Wed Nov 19 19:39:05 2003 @@ -110,6 +110,10 @@ implicit_exceptions = { 'getitem': [IndexError], } +class ImplicitExcValue: + def __repr__(self): + return 'implicitexc' +implicitexc = ImplicitExcValue() def make_op(name, symbol, arity, specialnames): if hasattr(FlowObjSpace, name): @@ -161,7 +165,8 @@ context = self.getexecutioncontext() outcome = context.guessbool(w_curexc, [None] + exceptions) if outcome is not None: - raise OperationError(self.wrap(outcome), self.w_None) + raise OperationError(self.wrap(outcome), + self.wrap(implicitexc)) return w_result setattr(FlowObjSpace, name, generic_operator) Modified: pypy/trunk/src/pypy/translator/gencl.py ============================================================================== --- pypy/trunk/src/pypy/translator/gencl.py (original) +++ pypy/trunk/src/pypy/translator/gencl.py Wed Nov 19 19:39:05 2003 @@ -1,5 +1,6 @@ import autopath from pypy.objspace.flow.model import * +from pypy.objspace.flow.objspace import implicitexc from pypy.translator.annrpython import RPythonAnnotator from pypy.translator.simplify import simplify_graph @@ -194,6 +195,8 @@ return val elif isinstance(val, type(Exception)) and issubclass(val, Exception): return "'%s" % val.__name__ + elif val is implicitexc: + return "'implicitexc" else: return "#<%r>" % (val,) def emitcode(self): Modified: pypy/trunk/src/pypy/translator/simplify.py ============================================================================== --- pypy/trunk/src/pypy/translator/simplify.py (original) +++ pypy/trunk/src/pypy/translator/simplify.py Wed Nov 19 19:39:05 2003 @@ -2,6 +2,7 @@ """ from pypy.objspace.flow.model import * +from pypy.objspace.flow.objspace import implicitexc def eliminate_empty_blocks(graph): """Eliminate basic blocks that do not contain any operations. @@ -60,8 +61,19 @@ visit(exit) traverse(visit, graph) +def remove_implicit_exceptions(graph): + def visit(link): + if isinstance(link, Link) and link in link.prevblock.exits: + if (link.args == [Constant(implicitexc)] and not link.target.exits + and hasattr(link.target, 'exc_type')): + lst = list(link.prevblock.exits) + lst.remove(link) + link.prevblock.exits = tuple(lst) + traverse(visit, graph) + def simplify_graph(graph): """Apply all the existing optimisations to the graph.""" eliminate_empty_blocks(graph) join_blocks(graph) + remove_implicit_exceptions(graph) return graph From arigo at codespeak.net Thu Nov 20 12:12:10 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 20 Nov 2003 12:12:10 +0100 (MET) Subject: [pypy-svn] rev 2231 - pypy/trunk/src/pypy/translator Message-ID: <20031120111210.075475B45F@thoth.codespeak.net> Author: arigo Date: Thu Nov 20 12:12:09 2003 New Revision: 2231 Modified: pypy/trunk/src/pypy/translator/simplify.py Log: Fixed bug in remove_implicit_exceptions(). Modified: pypy/trunk/src/pypy/translator/simplify.py ============================================================================== --- pypy/trunk/src/pypy/translator/simplify.py (original) +++ pypy/trunk/src/pypy/translator/simplify.py Thu Nov 20 12:12:09 2003 @@ -64,8 +64,12 @@ def remove_implicit_exceptions(graph): def visit(link): if isinstance(link, Link) and link in link.prevblock.exits: - if (link.args == [Constant(implicitexc)] and not link.target.exits - and hasattr(link.target, 'exc_type')): + if (isinstance(link.exitcase, type(Exception)) and + issubclass(link.exitcase, Exception) and + link.args == [Constant(implicitexc)] and + len(link.target.exits) == 0 and + hasattr(link.target, 'exc_type')): + # remove direct links to implicit exception return blocks lst = list(link.prevblock.exits) lst.remove(link) link.prevblock.exits = tuple(lst) @@ -74,6 +78,6 @@ def simplify_graph(graph): """Apply all the existing optimisations to the graph.""" eliminate_empty_blocks(graph) - join_blocks(graph) remove_implicit_exceptions(graph) + join_blocks(graph) return graph From arigo at codespeak.net Thu Nov 20 12:56:38 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 20 Nov 2003 12:56:38 +0100 (MET) Subject: [pypy-svn] rev 2232 - pypy/trunk/src/pypy/objspace/flow Message-ID: <20031120115638.983105B457@thoth.codespeak.net> Author: arigo Date: Thu Nov 20 12:56:37 2003 New Revision: 2232 Modified: pypy/trunk/src/pypy/objspace/flow/objspace.py Log: Preventing calls with constant arguments to happen in advance. And a minor optimization. Modified: pypy/trunk/src/pypy/objspace/flow/objspace.py ============================================================================== --- pypy/trunk/src/pypy/objspace/flow/objspace.py (original) +++ pypy/trunk/src/pypy/objspace/flow/objspace.py Thu Nov 20 12:56:37 2003 @@ -1,5 +1,5 @@ # ______________________________________________________________________ -import sys, operator +import sys, operator, types import pypy from pypy.interpreter.baseobjspace import ObjSpace, NoValue from pypy.interpreter.pycode import PyCode @@ -121,9 +121,9 @@ op = getattr(operator, name, None) if not op: - if name == 'call': - op = apply - elif name == 'issubtype': + #if name == 'call': + # op = apply + if name == 'issubtype': op = issubclass elif name == 'id': op = id @@ -134,16 +134,16 @@ def generic_operator(self, *args_w): assert len(args_w) == arity, name+" got the wrong number of arguments" - args = [] - for w_arg in args_w: - try: - arg = self.unwrap(w_arg) - except UnwrapException: - break + if op: + args = [] + for w_arg in args_w: + try: + arg = self.unwrap(w_arg) + except UnwrapException: + break + else: + args.append(arg) else: - args.append(arg) - else: - if op: # All arguments are constants: call the operator now #print >> sys.stderr, 'Constant operation', op try: From arigo at codespeak.net Thu Nov 20 13:04:28 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 20 Nov 2003 13:04:28 +0100 (MET) Subject: [pypy-svn] rev 2233 - in pypy/trunk/src/pypy/translator: . test tool Message-ID: <20031120120428.A15CC5B457@thoth.codespeak.net> Author: arigo Date: Thu Nov 20 13:04:27 2003 New Revision: 2233 Modified: pypy/trunk/src/pypy/translator/annrpython.py pypy/trunk/src/pypy/translator/test/snippet.py pypy/trunk/src/pypy/translator/test/test_annrpython.py pypy/trunk/src/pypy/translator/tool/make_dot.py pypy/trunk/src/pypy/translator/transform.py pypy/trunk/src/pypy/translator/translator.py Log: Successful analysis of the factorial! M pypy/translator/translator.py Translator is now the class that handles cross-function analysis. Some cleanup is required but it seems to work. M pypy/translator/test/snippet.py M pypy/translator/test/test_annrpython.py Factorial function and test. M pypy/translator/annrpython.py The 'call' operation invokes the Translator, which makes a control flow graph of the called function and handle it back to the Annotator's list of pending blocks. M pypy/translator/transform.py Moved some code into annrpython.py. M pypy/translator/tool/make_dot.py t.gv() in translator.py displays the graph of all functions in the same ps file (until we have some more interactive way of walking around). Modified: pypy/trunk/src/pypy/translator/annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/annrpython.py (original) +++ pypy/trunk/src/pypy/translator/annrpython.py Thu Nov 20 13:04:27 2003 @@ -1,5 +1,6 @@ from __future__ import generators +from types import FunctionType from pypy.translator.annheap import AnnotationHeap, Transaction from pypy.translator.annotation import XCell, XConstant, nothingyet from pypy.translator.annotation import Annotation @@ -7,15 +8,20 @@ from pypy.objspace.flow.model import SpaceOperation +class AnnotatorError(Exception): + pass + + class RPythonAnnotator: """Block annotator for RPython. See description in doc/transation/annotation.txt.""" - def __init__(self): + def __init__(self, translator=None): self.heap = AnnotationHeap() self.pendingblocks = [] # list of (block, list-of-XCells) self.bindings = {} # map Variables/Constants to XCells/XConstants self.annotated = {} # set of blocks already seen + self.translator = translator # build default annotations t = self.transaction() self.any_immutable = XCell() @@ -52,12 +58,24 @@ def complete(self): """Process pending blocks until none is left.""" + delayed = [] while self.pendingblocks: # XXX don't know if it is better to pop from the head or the tail. # let's do it breadth-first and pop from the head (oldest first). # that's more stacklessy. block, cells = self.pendingblocks.pop(0) - self.processblock(block, cells) + try: + self.processblock(block, cells) + except DelayAnnotation: + delayed.append((block, cells)) + else: + # when processblock succeed, i.e. when the analysis progress, + # we can tentatively re-schedlue the delayed blocks. + self.pendingblocks += delayed + del delayed[:] + if delayed: + raise AnnotatorError('%d delayed and suspended block(s)' % + len(delayed)) def binding(self, arg): "XCell or XConstant corresponding to the given Variable or Constant." @@ -243,10 +261,34 @@ if type1 in (list, tuple) and type2 is slice: t.set_type(result, type1) + def decode_simple_call(self, varargs_cell, varkwds_cell, t): + len_cell = t.get('len', [varargs_cell]) + if not isinstance(len_cell, XConstant): + return None + nbargs = len_cell.value + arg_cells = [t.get('getitem', [varargs_cell, self.constant(j)]) + for j in range(nbargs)] + if None in arg_cells: + return None + len_cell = t.get('len', [varkwds_cell]) + if not isinstance(len_cell, XConstant): + return None + nbkwds = len_cell.value + if nbkwds != 0: + return None + return arg_cells + def consider_op_call(self, (func,varargs,kwargs), result, t): if not isinstance(func, XConstant): return func = func.value + if isinstance(func, FunctionType) and self.translator: + args = self.decode_simple_call(varargs, kwargs, t) + if args is not None: + result_cell = self.translator.consider_call(self, func, args) + if result_cell is nothingyet: + raise DelayAnnotation + # XXX: generalize this later if func is range: t.set_type(result, list) @@ -266,3 +308,6 @@ class CannotSimplify(Exception): pass + +class DelayAnnotation(Exception): + pass Modified: pypy/trunk/src/pypy/translator/test/snippet.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/snippet.py (original) +++ pypy/trunk/src/pypy/translator/test/snippet.py Thu Nov 20 13:04:27 2003 @@ -251,3 +251,9 @@ finally: x = 8 return x + +def factorial(n): + if n <= 1: + return 1 + else: + return n * factorial(n-1) Modified: pypy/trunk/src/pypy/translator/test/test_annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/test_annrpython.py (original) +++ pypy/trunk/src/pypy/translator/test/test_annrpython.py Thu Nov 20 13:04:27 2003 @@ -4,6 +4,7 @@ from pypy.tool.udir import udir from pypy.translator.annrpython import RPythonAnnotator +from pypy.translator.translator import Translator from pypy.objspace.flow.model import * from pypy.translator.test import snippet @@ -131,6 +132,16 @@ self.failIf(item_cell is None) self.assertEquals(t.get_type(item_cell), int) + def test_factorial(self): + translator = Translator(snippet.factorial) + graph = translator.getflowgraph() + a = RPythonAnnotator(translator) + a.build_types(graph, [int]) + # result should be an integer + t = a.transaction() + end_cell = a.binding(graph.getreturnvar()) + self.assertEquals(t.get_type(end_cell), int) + def g(n): return [0,1,2,n] Modified: pypy/trunk/src/pypy/translator/tool/make_dot.py ============================================================================== --- pypy/trunk/src/pypy/translator/tool/make_dot.py (original) +++ pypy/trunk/src/pypy/translator/tool/make_dot.py Thu Nov 20 13:04:27 2003 @@ -129,16 +129,29 @@ def make_dot(graphname, graph, storedir=None, target='ps'): + return make_dot_graphs(graph.name, [(graphname, graph)], storedir, target) + +def make_dot_graphs(basefilename, graphs, storedir=None, target='ps'): from vpath.adapter.process import exec_cmd if storedir is None: storedir = udir dotgen = DotGen() - name = graph.name - dest = storedir.join('%s.dot' % name) + dest = storedir.join('%s.dot' % basefilename) #dest = storedir.dirname().join('%s.dot' % name) - source = dotgen.getdigraph(graphname, graph) + subgraphs = [] + basefilename = '_'+basefilename + names = {basefilename: True} + for graphname, graph in graphs: + if graphname in names: + i = 2 + while graphname + str(i) in names: + i += 1 + graphname = graphname + str(i) + names[graphname] = True + subgraphs.append(dotgen.getsubgraph(graphname, graph)) + source = dotgen.getgraph(basefilename, subgraphs) #print source dest.write(source) psdest = dest.newext(target) Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Thu Nov 20 13:04:27 2003 @@ -83,21 +83,9 @@ raise CannotSimplify varargs_cell = self.binding(op.args[1]) varkwds_cell = self.binding(op.args[2]) - - len_cell = t.get('len', [varargs_cell]) - if not isinstance(len_cell, XConstant): - raise CannotSimplify - nbargs = len_cell.value - arg_cells = [t.get('getitem', [varargs_cell, self.constant(j)]) - for j in range(nbargs)] - if None in arg_cells: - raise CannotSimplify - - len_cell = t.get('len', [varkwds_cell]) - if not isinstance(len_cell, XConstant): - raise CannotSimplify - nbkwds = len_cell.value - if nbkwds != 0: + arg_cells = self.decode_simple_call(varargs_cell, + varkwds_cell, t) + if arg_cells is None: raise CannotSimplify args = [self.reverse_binding(known_vars, c) for c in arg_cells] Modified: pypy/trunk/src/pypy/translator/translator.py ============================================================================== --- pypy/trunk/src/pypy/translator/translator.py (original) +++ pypy/trunk/src/pypy/translator/translator.py Thu Nov 20 13:04:27 2003 @@ -32,6 +32,7 @@ import autopath from pypy.objspace.flow.model import * +from pypy.translator.annotation import * from pypy.translator.annrpython import RPythonAnnotator from pypy.translator.simplify import simplify_graph from pypy.translator.genpyrex import GenPyrex @@ -46,70 +47,121 @@ def __init__(self, func): self.entrypoint = func + self.clear() + + def clear(self): + """Clear all annotations and all flow graphs.""" self.annotator = None - space = FlowObjSpace() - self.flowgraph = space.build_flow(func) + self.flowgraphs = {} # {function: graph} + self.functions = [] # the keys of self.flowgraphs, in creation order + self.getflowgraph() + + def getflowgraph(self, func=None): + """Get the flow graph for a function (default: the entry point).""" + func = func or self.entrypoint try: - import inspect - self.py_source = inspect.getsource(func) - except IOError: - # e.g. when func is defined interactively - self.py_source = "" - - def gv(self): - """Shows the control flow graph -- requires 'dot' and 'gv'.""" + graph = self.flowgraphs[func] + except KeyError: + space = FlowObjSpace() + graph = self.flowgraphs[func] = space.build_flow(func) + self.functions.append(func) + try: + import inspect + graph.source = inspect.getsource(func) + except IOError: + pass # e.g. when func is defined interactively + return graph + + def gv(self, func=None): + """Shows the control flow graph for a function (default: all) + -- requires 'dot' and 'gv'.""" import os - from pypy.translator.tool.make_dot import make_dot - dest = make_dot('dummy', self.flowgraph) + from pypy.translator.tool.make_dot import make_dot, make_dot_graphs + if func is None: + # show the graph of *all* functions at the same time + graphs = [] + for func in self.functions: + graph = self.getflowgraph(func) + graphs.append((graph.name, graph)) + dest = make_dot_graphs(self.entrypoint.__name__, graphs) + else: + graph = self.getflowgraph(func) + dest = make_dot(graph.name, graph) os.system('gv %s' % str(dest)) - def simplify(self): - """Simplifies the control flow graph.""" - self.flowgraph = simplify_graph(self.flowgraph) + def simplify(self, func=None): + """Simplifies the control flow graph (default: for all functions).""" + if func is None: + for func in self.flowgraphs.keys(): + self.simplify(func) + else: + graph = self.getflowgraph(func) + self.flowgraphs[func] = simplify_graph(graph) - def annotate(self, input_args_types): - """annotate(self, input_arg_types) -> Annotator + def annotate(self, input_args_types, func=None): + """annotate(self, input_arg_types[, func]) -> Annotator Provides type information of arguments. Returns annotator. """ - self.annotator = RPythonAnnotator() - self.annotator.build_types(self.flowgraph, input_args_types) + func = func or self.entrypoint + if self.annotator is None: + self.annotator = RPythonAnnotator(self) + graph = self.getflowgraph(func) + self.annotator.build_types(graph, input_args_types) return self.annotator - def source(self): + def source(self, func=None): """Returns original Python source. Returns for functions written while the interactive session. """ - return self.py_source + func = func or self.entrypoint + graph = self.getflowgraph(func) + return getattr(graph, 'source', '') - def pyrex(self, input_arg_types=None): - """pyrex(self, [input_arg_types]) -> Pyrex translation + def pyrex(self, input_arg_types=None, func=None): + """pyrex(self[, input_arg_types][, func]) -> Pyrex translation Returns Pyrex translation. If input_arg_types is provided, returns type annotated translation. Subsequent calls are not affected by this. """ - g = GenPyrex(self.flowgraph) - if input_arg_types is not None: - g.annotate(input_arg_types) - elif self.annotator: - g.setannotator(self.annotator) - return g.emitcode() + return self.generatecode(GenPyrex, input_arg_types, func) def cl(self, input_arg_types=None): - """cl(self, [input_arg_types]) -> Common Lisp translation + """cl(self[, input_arg_types][, func]) -> Common Lisp translation Returns Common Lisp translation. If input_arg_types is provided, returns type annotated translation. Subsequent calls are not affected by this. """ - g = GenCL(self.flowgraph) + return self.generatecode(GenCL, input_arg_types, func) + + def generatecode(self, gencls, input_arg_types, func): + if input_arg_types is None: + ann = self.annotator + else: + ann = RPythonAnnotator(self) + if func is None: + code = self.generatecode1(gencls, input_arg_types, + self.entrypoint, ann) + codes = [code] + for func in self.functions: + if func is not self.entrypoint: + code = self.generatecode1(gencls, None, func, ann) + codes.append(code) + return '\n\n#_________________\n\n'.join(codes) + else: + return self.generatecode1(gencls, input_arg_types, func, ann) + + def generatecode1(self, gencls, input_arg_types, func, ann): + graph = self.getflowgraph(func) + g = gencls(graph) if input_arg_types is not None: - g.annotate(input_arg_types) - elif self.annotator: - g.ann = self.annotator + ann.build_types(graph, input_arg_types) + if ann is not None: + g.setannotator(ann) return g.emitcode() def compile(self): @@ -127,10 +179,21 @@ """Calls underlying Python function.""" return self.entrypoint(*args) - def dis(self): + def dis(self, func=None): """Disassembles underlying Python function to bytecodes.""" from dis import dis - dis(self.entrypoint) + dis(func or self.entrypoint) + + def consider_call(self, ann, func, args): + graph = self.getflowgraph(func) + ann.addpendingblock(graph.startblock, args) + result_var = graph.getreturnvar() + try: + return ann.binding(result_var) + except KeyError: + # typical case for the 1st call, because addpendingblock() did + # not actually start the analysis of the called function yet. + return nothingyet if __name__ == '__main__': From arigo at codespeak.net Thu Nov 20 13:14:06 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 20 Nov 2003 13:14:06 +0100 (MET) Subject: [pypy-svn] rev 2234 - pypy/trunk/src/pypy/translator Message-ID: <20031120121406.9575B5B457@thoth.codespeak.net> Author: arigo Date: Thu Nov 20 13:14:05 2003 New Revision: 2234 Modified: pypy/trunk/src/pypy/translator/annrpython.py Log: Missing code: doing something with the result of 'call' operations. Now I have to investigate how it could possibly have passed the tests before... Modified: pypy/trunk/src/pypy/translator/annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/annrpython.py (original) +++ pypy/trunk/src/pypy/translator/annrpython.py Thu Nov 20 13:14:05 2003 @@ -172,7 +172,11 @@ resultcell = self.bindnew(op.result) consider_meth = getattr(self,'consider_op_'+op.opname,None) if consider_meth is not None: - consider_meth(argcells, resultcell, self.transaction()) + newresult = consider_meth(argcells, resultcell, self.transaction()) + # XXX not too clean: most consider_op_xxx() implicitely return None, + # but consider_op_call() needs to return an explicit resultcell. + if newresult is not None: + self.bindings[op.result] = newresult def consider_op_add(self, (arg1,arg2), result, t): type1 = t.get_type(arg1) @@ -288,6 +292,7 @@ result_cell = self.translator.consider_call(self, func, args) if result_cell is nothingyet: raise DelayAnnotation + return result_cell # XXX: generalize this later if func is range: From arigo at codespeak.net Thu Nov 20 13:51:37 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 20 Nov 2003 13:51:37 +0100 (MET) Subject: [pypy-svn] rev 2235 - pypy/trunk/src/pypy/translator Message-ID: <20031120125137.B550E5B457@thoth.codespeak.net> Author: arigo Date: Thu Nov 20 13:51:37 2003 New Revision: 2235 Modified: pypy/trunk/src/pypy/translator/annrpython.py Log: Now this looks more like the correct algorithm for re-flowing analysis through delayed blocks. Modified: pypy/trunk/src/pypy/translator/annrpython.py ============================================================================== --- pypy/trunk/src/pypy/translator/annrpython.py (original) +++ pypy/trunk/src/pypy/translator/annrpython.py Thu Nov 20 13:51:37 2003 @@ -19,6 +19,7 @@ def __init__(self, translator=None): self.heap = AnnotationHeap() self.pendingblocks = [] # list of (block, list-of-XCells) + self.delayedblocks = [] # list of blocked blocks self.bindings = {} # map Variables/Constants to XCells/XConstants self.annotated = {} # set of blocks already seen self.translator = translator @@ -58,23 +59,14 @@ def complete(self): """Process pending blocks until none is left.""" - delayed = [] while self.pendingblocks: # XXX don't know if it is better to pop from the head or the tail. # let's do it breadth-first and pop from the head (oldest first). # that's more stacklessy. block, cells = self.pendingblocks.pop(0) - try: - self.processblock(block, cells) - except DelayAnnotation: - delayed.append((block, cells)) - else: - # when processblock succeed, i.e. when the analysis progress, - # we can tentatively re-schedlue the delayed blocks. - self.pendingblocks += delayed - del delayed[:] - if delayed: - raise AnnotatorError('%d delayed and suspended block(s)' % + self.processblock(block, cells) + if self.delayedblocks: + raise AnnotatorError('%d block(s) are still blocked' % len(delayed)) def binding(self, arg): @@ -136,28 +128,58 @@ #___ flowing annotations in blocks _____________________ def processblock(self, block, cells): + # Important: this is not called recursively. + # self.flowin() can only issue calls to self.addpendingblock(). + # The analysis of a block can be in three states: + # * block not in self.annotated: + # never seen the block. + # * self.annotated[block] == False: + # the input variables of the block are in self.bindings but we + # still have to consider all the operations in the block. + # * self.annotated[block] == True: + # analysis done (at least until we find we must generalize the + # input variables). + #print '* processblock', block, cells if block not in self.annotated: - self.annotated[block] = True - self.flowin(block, cells) - else: - # already seen; merge each of the block's input variable - oldcells = [] - newcells = [] - for a, cell2 in zip(block.inputargs, cells): - cell1 = self.bindings[a] # old binding - oldcells.append(cell1) - newcells.append(self.heap.merge(cell1, cell2)) - #print '** oldcells = ', oldcells - #print '** newcells = ', newcells - # re-flowin unless the newcells are equal to the oldcells - if newcells != oldcells: - self.flowin(block, newcells) + self.bindinputargs(block, cells) + elif cells is not None: + self.mergeinputargs(block, cells) + if not self.annotated[block]: + try: + self.flowin(block) + except DelayAnnotation: + self.delayedblocks.append(block) # failed, hopefully temporarily + else: + self.annotated[block] = True + # When flowin succeeds, i.e. when the analysis progress, + # we can tentatively re-schedlue the delayed blocks. + for block in self.delayedblocks: + self.pendingblocks.append((block, None)) + del self.delayedblocks[:] - def flowin(self, block, inputcells): - #print '...' + def bindinputargs(self, block, inputcells): + # Create the initial bindings for the input args of a block. for a, cell in zip(block.inputargs, inputcells): self.bindings[a] = cell + self.annotated[block] = False # must flowin. + + def mergeinputargs(self, block, inputcells): + # Merge the new 'cells' with each of the block's existing input + # variables. + oldcells = [] + newcells = [] + for a, cell2 in zip(block.inputargs, inputcells): + cell1 = self.bindings[a] # old binding + oldcells.append(cell1) + newcells.append(self.heap.merge(cell1, cell2)) + #print '** oldcells = ', oldcells + #print '** newcells = ', newcells + # if the merged cells changed, we must redo the analysis + if newcells != oldcells: + self.bindinputargs(block, newcells) + + def flowin(self, block): for op in block.operations: self.consider_op(op) for link in block.exits: @@ -172,11 +194,7 @@ resultcell = self.bindnew(op.result) consider_meth = getattr(self,'consider_op_'+op.opname,None) if consider_meth is not None: - newresult = consider_meth(argcells, resultcell, self.transaction()) - # XXX not too clean: most consider_op_xxx() implicitely return None, - # but consider_op_call() needs to return an explicit resultcell. - if newresult is not None: - self.bindings[op.result] = newresult + consider_meth(argcells, resultcell, self.transaction()) def consider_op_add(self, (arg1,arg2), result, t): type1 = t.get_type(arg1) @@ -191,6 +209,14 @@ t.set_type(result, list) # XXX propagate information about the type of the elements + def consider_op_mul(self, (arg1,arg2), result, t): + type1 = t.get_type(arg1) + type2 = t.get_type(arg2) + if type1 is int and type2 is int: + t.set_type(result, int) + elif type1 in (int, long) and type2 in (int, long): + t.set_type(result, long) + def consider_op_inplace_add(self, (arg1,arg2), result, t): type1 = t.get_type(arg1) type2 = t.get_type(arg1) @@ -292,7 +318,15 @@ result_cell = self.translator.consider_call(self, func, args) if result_cell is nothingyet: raise DelayAnnotation - return result_cell + # 'result' is made shared with 'result_cell'. This has the + # effect that even if result_cell is actually an XConstant, + # result stays an XCell, but the annotations about the constant + # are also appliable to result. This is bad because it means + # functions returning constants won't propagate the constant + # but only e.g. its type. This is needed at this point because + # XConstants are not too well supported in the forward_deps + # lists: forward_deps cannot downgrade XConstant to XCell. + result.share(result_cell) # XXX: generalize this later if func is range: From arigo at codespeak.net Thu Nov 20 15:00:02 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 20 Nov 2003 15:00:02 +0100 (MET) Subject: [pypy-svn] rev 2236 - pypy/trunk/src/pypy/translator Message-ID: <20031120140002.DEF525B90F@thoth.codespeak.net> Author: arigo Date: Thu Nov 20 15:00:02 2003 New Revision: 2236 Modified: pypy/trunk/src/pypy/translator/genpyrex.py Log: Quick hack to allow Pyrexed functions to call other Pyrexed functions. Modified: pypy/trunk/src/pypy/translator/genpyrex.py ============================================================================== --- pypy/trunk/src/pypy/translator/genpyrex.py (original) +++ pypy/trunk/src/pypy/translator/genpyrex.py Thu Nov 20 15:00:02 2003 @@ -249,8 +249,8 @@ except AttributeError: pass else: - if __builtins__.get(name) is value: - return name # built-in functions represented as their name only + if callable(value): + return name # functions represented as their name only if isinstance(value, int): value = int(value) # cast subclasses of int (i.e. bools) to ints return repr(value) From hpk at codespeak.net Sat Nov 22 14:14:06 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sat, 22 Nov 2003 14:14:06 +0100 (MET) Subject: [pypy-svn] rev 2244 - pypy/trunk/src/pypy/translator Message-ID: <20031122131406.D0D355B935@thoth.codespeak.net> Author: hpk Date: Sat Nov 22 14:14:05 2003 New Revision: 2244 Modified: pypy/trunk/src/pypy/translator/annotation.py Log: removed somewhat confusing name attribute from XCell (it was only needed for debugging purposes) Modified: pypy/trunk/src/pypy/translator/annotation.py ============================================================================== --- pypy/trunk/src/pypy/translator/annotation.py (original) +++ pypy/trunk/src/pypy/translator/annotation.py Sat Nov 22 14:14:05 2003 @@ -41,26 +41,30 @@ return tuple(lst) +def debugname(xcell, name=None, _seen = {}): + """ return a simple name for an xcell. """ + try: + return _seen[id(xcell)] + except KeyError: + if name is None: + name = "X%d" % len(seen) + _seen[id(xcell)] = name + return name + class XCell: """A placeholder for a heap object contained in an AnnotationHeap. It represents an object that will actually appear at run-time in the heap. XCells are the arguments and return value of Annotations.""" - counter = 0 - # Multiple XCells can be "shared"; a group of shared cells # act essentially like a single cell (they become all equal). - def __init__(self, name=None): - if not name: - name = 'X%d' % XCell.counter - XCell.counter += 1 - self.name = name + def __init__(self): self.shared = [] # list of weakrefs to XCells # defining a group of shared cells. def __repr__(self): - names = [cell.name for cell in self.cellsingroup()] + names = [debugname(cell) for cell in self.cellsingroup()] names.sort() return '=='.join(names) @@ -130,4 +134,6 @@ # This is specified by using nothingyet instead of a real XCell(). # Conversely, *no* annotation stands for any object. -nothingyet = XCell('nothingyet') +nothingyet = XCell() +debugname(nothingyet, 'nothingyet') + From hpk at codespeak.net Sat Nov 22 15:39:21 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sat, 22 Nov 2003 15:39:21 +0100 (MET) Subject: [pypy-svn] rev 2245 - pypy/trunk/src/pypy/translator Message-ID: <20031122143921.884E95B935@thoth.codespeak.net> Author: hpk Date: Sat Nov 22 15:39:20 2003 New Revision: 2245 Removed: pypy/trunk/src/pypy/translator/autopath.py Modified: pypy/trunk/src/pypy/translator/gencl.py pypy/trunk/src/pypy/translator/transform.py pypy/trunk/src/pypy/translator/translator.py Log: removed autopath.py and references because there are no real direct script executions here. Deleted: /pypy/trunk/src/pypy/translator/autopath.py ============================================================================== --- /pypy/trunk/src/pypy/translator/autopath.py Sat Nov 22 15:39:20 2003 +++ (empty file) @@ -1,78 +0,0 @@ -""" -self cloning, automatic path configuration - -copy this into any subdirectory of pypy from which scripts need -to be run, typically all of the test subdirs. -The idea is that any such script simply issues - - import autopath - -and this will make sure that the parent directory containing "pypy" -is in sys.path. - -If you modify the master "autopath.py" version (in pypy/tool/autopath.py) -you can directly run it which will copy itself on all autopath.py files -it finds under the pypy root directory. - -This module always provides these attributes: - - pypydir pypy root directory path - this_dir directory where this autopath.py resides - -""" - - -def __dirinfo(part): - """ return (partdir, this_dir) and insert parent of partdir - into sys.path. If the parent directories dont have the part - an EnvironmentError is raised.""" - - import sys, os - try: - head = this_dir = os.path.abspath(os.path.dirname(__file__)) - except NameError: - head = this_dir = os.path.abspath(os.path.dirname(sys.argv[0])) - - while head: - partdir = head - head, tail = os.path.split(head) - if tail == part: - sys.path = [p for p in sys.path if not p.startswith(head)] - if head not in sys.path: - sys.path.insert(0, head) - return partdir, this_dir - - raise EnvironmentError, "'%s' missing in '%r'" % (pathpart,this_path) - -def __clone(): - """ clone master version of autopath.py into all subdirs """ - from os.path import join, walk - if not this_dir.endswith(join('pypy','tool')): - raise EnvironmentError("can only clone master version " - "'%s'" % join(pypydir, 'tool',_myname)) - - - def sync_walker(arg, dirname, fnames): - if _myname in fnames: - fn = join(dirname, _myname) - f = open(fn, 'rwb+') - try: - if f.read() == arg: - print "checkok", fn - else: - print "syncing", fn - f = open(fn, 'w') - f.write(arg) - finally: - f.close() - s = open(join(pypydir, 'tool', _myname), 'rb').read() - walk(pypydir, sync_walker, s) - -_myname = 'autopath.py' - -# set guaranteed attributes - -pypydir, this_dir = __dirinfo('pypy') - -if __name__ == '__main__': - __clone() Modified: pypy/trunk/src/pypy/translator/gencl.py ============================================================================== --- pypy/trunk/src/pypy/translator/gencl.py (original) +++ pypy/trunk/src/pypy/translator/gencl.py Sat Nov 22 15:39:20 2003 @@ -1,4 +1,3 @@ -import autopath from pypy.objspace.flow.model import * from pypy.objspace.flow.objspace import implicitexc from pypy.translator.annrpython import RPythonAnnotator Modified: pypy/trunk/src/pypy/translator/transform.py ============================================================================== --- pypy/trunk/src/pypy/translator/transform.py (original) +++ pypy/trunk/src/pypy/translator/transform.py Sat Nov 22 15:39:20 2003 @@ -4,7 +4,6 @@ transformation may introduce new space operation. """ -import autopath import types from pypy.objspace.flow.model import SpaceOperation from pypy.translator.annotation import XCell, XConstant Modified: pypy/trunk/src/pypy/translator/translator.py ============================================================================== --- pypy/trunk/src/pypy/translator/translator.py (original) +++ pypy/trunk/src/pypy/translator/translator.py Sat Nov 22 15:39:20 2003 @@ -28,8 +28,7 @@ Try dir(test) for list of current snippets. """ - -import autopath +import test.autopath from pypy.objspace.flow.model import * from pypy.translator.annotation import * From hpk at codespeak.net Sat Nov 22 21:32:59 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sat, 22 Nov 2003 21:32:59 +0100 (MET) Subject: [pypy-svn] rev 2246 - in pypy/trunk/src/pypy/annotation: . test Message-ID: <20031122203259.C208A5B903@thoth.codespeak.net> Author: hpk Date: Sat Nov 22 21:32:59 2003 New Revision: 2246 Added: pypy/trunk/src/pypy/annotation/ pypy/trunk/src/pypy/annotation/test/ pypy/trunk/src/pypy/annotation/test/test_annset.py Log: added a new annotation directory with no code but only a first test for a new in-discussion refactoring of the current translator package. The basic idea is to - instead of XCells with complicated sharing-properties we want to have SomeValue-instances which have no state on themselves. Properties like 'sharedness' are then managed by an AnnotationSet. - an AnnotationSet will hold a list of Annotations with an interface that allows sharing of SomeValue's in specific ways. (see the test in the diff). - we want an Annotation instance to specify a 'predicate' and the input/result arguments. A predicate can actually represent a space operation or some other "internal" operation. Therefore XConstant disappears and "constant meaning" is now represented as an internal operation "is-a-constant-of-value-x" -- there is one such operation for each x. e.g. Annotation(op.constant(5), somevalue) allows somevalue to still be a regular SomeValue instance and we don't have to distinguish between different special kinds of e.g. XCell/XConstants any more. Added: pypy/trunk/src/pypy/annotation/test/test_annset.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/annotation/test/test_annset.py Sat Nov 22 21:32:59 2003 @@ -0,0 +1,65 @@ + +import autopath +from pypy.tool import test + +from pypy.annotation.model import Annotation, SomeValue + +# to avoid quoting of strings +class _op: + def __getattr__(self, name): + return name +op = _op() + +class TestAnnotationSet(test.IntTestCase): + + def test_shared_values(self): + c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() + a = AnnotationSet() + self.assert_(a.isshared(c1, c1)) + self.failIf(a.isshared(c1, c2)) + a.setshared(c1, c2) + self.assert_(a.isshared(c1, c2)) + self.assert_(a.isshared(c2, c1)) + self.assertEquals(a.tempid(c1), a.tempid(c2)) + self.failIfEqual(a.tempid(c1), a.tempid(c3)) + self.failIf(a.isshared(c1, c3)) + a.setshared(c2, c3) + self.assert_(a.isshared(c1, c3)) + self.assert_(a.isshared(c2, c3)) + self.assert_(a.isshared(c3, c1)) + self.assertEquals(a.tempid(c1), a.tempid(c3)) + + def test_shared_values_nomatch(self): + c1,c2 = SomeValue(), SomeValue() + a = AnnotationSet() + id1 = a.tempid(c1) + id2 = a.tempid(c2) + self.assertNotEquals(id1, id2) + + def test_query_one_annotation_arg(self): + c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() + lst = [Annotation(op.add, c1, c3, c2)] + a = AnnotationSet(lst) + c = a.query(Annotation(op.add, c1, c3, QUERYARG)) + self.assertEquals(c, [c2]) + c = a.query(Annotation(op.add, c1, QUERYARG, c2)) + self.assertEquals(c, [c3]) + c = a.query(Annotation(op.add, QUERYARG, c3, c2)) + self.assertEquals(c, [c1]) + + c = a.query(Annotation(op.add, QUERYARG, c1, c2)) + self.assertEquals(c, []) + + def test_query_multiple_annotations(self): + c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() + lst = [ + Annotation(op.add, c1, c3, c2), + Annotation(op.something, c2, c3) + ] + a = AnnotationSet(lst) + c = a.query(Annotation(op.add, c1, c3, QUERYARG), + Annotation(op.something, QUERYARG, c3)) + self.assertEquals(c, [c2]) + +if __name__ == '__main__': + test.main() From hpk at codespeak.net Tue Nov 25 11:43:25 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Tue, 25 Nov 2003 11:43:25 +0100 (MET) Subject: [pypy-svn] rev 2260 - in pypy/trunk/src/pypy: . objspace/flow translatortranslator/test Message-ID: <20031125104325.0D4CD5B9A1@thoth.codespeak.net> Author: hpk Date: Tue Nov 25 11:43:25 2003 New Revision: 2260 Removed: pypy/trunk/src/pypy/objspace/flow/wrapper.py Modified: pypy/trunk/src/pypy/TODO pypy/trunk/src/pypy/translator/annotation.py pypy/trunk/src/pypy/translator/test/snippet.py pypy/trunk/src/pypy/translator/test/test_pyrextrans.py Log: - got rid of the stale 'wrapper.py' file. - added a 'polybranch' test, i.e. a test where the two if-branches produce different types for the same binding. Current annotation/flowobjspace will not result in types for this case, currently. - updated TODO Modified: pypy/trunk/src/pypy/TODO ============================================================================== --- pypy/trunk/src/pypy/TODO (original) +++ pypy/trunk/src/pypy/TODO Tue Nov 25 11:43:25 2003 @@ -1,6 +1,3 @@ -Task: review and implement the annotation procedure described in - doc/translation/annotation.txt - Task: update pypy homepage Task: add more builtins! (like e.g. 'dir') Deleted: /pypy/trunk/src/pypy/objspace/flow/wrapper.py ============================================================================== --- /pypy/trunk/src/pypy/objspace/flow/wrapper.py Tue Nov 25 11:43:25 2003 +++ (empty file) @@ -1,81 +0,0 @@ -# ______________________________________________________________________ -"""Wrapper objects for the control flow analysis object space.""" -# ______________________________________________________________________ - -# This is kinda a hack, but at the same time, I don't see why this was defined -# in the object space module in the annotation object space. - -from pypy.translator.flowmodel import Variable, Constant - -class UnwrapException(Exception): - pass - -# ______________________________________________________________________ - -class W_Object(object): - """Abstract base class. do not instantiate.""" - - force = None # See cloningcontext.py - - def __new__(cls, *args, **kwd): - assert cls is not W_Object - return object.__new__(cls) - - def __init__(self): - pass - - def __repr__(self): - s = self.argsrepr() - name = getattr(self, "symname", "") - if not name: - name = self.__class__.__name__ - if len(s) > 100: - s = s[:25] + "..." + s[-25:] - return "%s(%s)" % (name, s) - - def argsrepr(self): - return "" - - def unwrap(self): - # XXX Somehow importing this at module level doesn't work - raise UnwrapException(self) - -# ______________________________________________________________________ - -class W_Variable(W_Object, Variable): - counter = 0 - symname = "var" - - def __init__(self): - Variable.__init__(self, 'v%d' % W_Variable.counter) - W_Variable.counter += 1 - - def argsrepr(self): - return self.pseudoname - -# ______________________________________________________________________ - -class W_Constant(W_Object, Constant): - """A specific constant value.""" - - symname = "const" - def __init__(self, value): - Constant.__init__(self, value) - - def argsrepr(self): - return repr(self.value) - - def unwrap(self): - return self.value - - def __len__(self): - return len(self.value) - - def __getitem__(self, key): - return self.value[key] - - def __setitem__(self, key, value): - self.value[key] = value - -# ______________________________________________________________________ -# End of wrapper.py Modified: pypy/trunk/src/pypy/translator/annotation.py ============================================================================== --- pypy/trunk/src/pypy/translator/annotation.py (original) +++ pypy/trunk/src/pypy/translator/annotation.py Tue Nov 25 11:43:25 2003 @@ -40,7 +40,6 @@ lst += [arg.temporarykey() for arg in self.args] return tuple(lst) - def debugname(xcell, name=None, _seen = {}): """ return a simple name for an xcell. """ try: @@ -107,7 +106,6 @@ c.shared = lst1 lst1.append(s) - class XConstant(XCell): """A fully determined XCell. For immutable constants.""" Modified: pypy/trunk/src/pypy/translator/test/snippet.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/snippet.py (original) +++ pypy/trunk/src/pypy/translator/test/snippet.py Tue Nov 25 11:43:25 2003 @@ -206,6 +206,15 @@ bitmask += 1 return powerset +def poly_branch(x): + if x: + y = [1,2,3] + else: + y = ['a','b','c'] + + z = y + return z*2 + def s_and(x, y): if x and y: return 'yes' Modified: pypy/trunk/src/pypy/translator/test/test_pyrextrans.py ============================================================================== --- pypy/trunk/src/pypy/translator/test/test_pyrextrans.py (original) +++ pypy/trunk/src/pypy/translator/test/test_pyrextrans.py Tue Nov 25 11:43:25 2003 @@ -72,6 +72,11 @@ half = self.build_cfunc(t.half_of_n) self.assertEquals(half(10), 5) + def test_poly_branch(self): + poly_branch = self.build_cfunc(t.poly_branch) + self.assertEquals(poly_branch(10), [1,2,3]*2) + self.assertEquals(poly_branch(0), ['a','b','c']*2) + def test_and(self): sand = self.build_cfunc(t.s_and) self.assertEquals(sand(5, 6), "yes") From hpk at codespeak.net Wed Nov 26 13:18:52 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Wed, 26 Nov 2003 13:18:52 +0100 (MET) Subject: [pypy-svn] rev 2266 - pypy/trunk/src/pypy/annotation Message-ID: <20031126121852.49EC05A975@thoth.codespeak.net> Author: hpk Date: Wed Nov 26 13:18:51 2003 New Revision: 2266 Added: pypy/trunk/src/pypy/annotation/README.txt Log: an introduction to the new annotation design (armin and holger) Added: pypy/trunk/src/pypy/annotation/README.txt ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/annotation/README.txt Wed Nov 26 13:18:51 2003 @@ -0,0 +1,39 @@ +guiding design/implementation ideas +----------------------------------- + +Annotation aims at providing type inference to RPython programs. +Annotation works from the flowmodel of a program. (A flowmodel +can be viewed as an information-preserving 'dead code' representation +of a program.) + +Annotation mainly deals with connecting interesting information +to SomeValue's which are instances of a very simple class:: + + class SomeValue: + pass + +A SomeValue represents a possible state within a program. +Any information about the type or more specific information +like being a constant is done by an Annotation. E.g. the +an int-type annotation is expressed like this: + + Annotation(op.type, someval1, someval2) + Annotation(op.constant(int), someval2) + +Note especially how the constant-ness of someval2 encodes the +information 'someval2 is exactly int'. SomeValue's are values on which +little is known by default, unless Annotation's are used to restrict +them. + +Keep in mind that the more Annotation's you have on SomeValue, the more +restricted it is, i.e. the less real values it can represent. A newly +created SomeValue() has no annotation by default, i.e. can represent +anything at all. At the other extreme, there is a special +'blackholevalue' that behaves as if it had all Annotation's set on it; +it stands for an impossible, non-existent value (because all these +Annotations are contradictory). The name 'blackholevalue' reminds you +that during type inference SomeValue's start with a lot of annotations +(possibly as a 'blackholevalue'), and annotations are killed -- less +annotations, more possibilities. + +Annotations are stored in a global list, which is an AnnotationSet instance. AnnotationSet provides (via Transactions) methods to query, add and kill annotations. It also manages "sharing": two different SomeValue's can be later found to be identical (in the Python sense of "is"), and the AnnotationSet can be taught about this. From hpk at codespeak.net Wed Nov 26 15:18:24 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Wed, 26 Nov 2003 15:18:24 +0100 (MET) Subject: [pypy-svn] rev 2267 - in pypy/trunk/src/pypy/annotation: . test Message-ID: <20031126141824.D4EFC5A975@thoth.codespeak.net> Author: hpk Date: Wed Nov 26 15:18:24 2003 New Revision: 2267 Added: pypy/trunk/src/pypy/annotation/__init__.py pypy/trunk/src/pypy/annotation/annset.py (contents, props changed) - copied, changed from rev 2265, pypy/trunk/src/pypy/translator/annheap.py pypy/trunk/src/pypy/annotation/model.py pypy/trunk/src/pypy/annotation/test/autopath.py (props changed) - copied unchanged from rev 2265, pypy/trunk/src/pypy/translator/test/autopath.py Modified: pypy/trunk/src/pypy/annotation/test/test_annset.py Log: intermediate checkin (armin, holger) the annset-tests pass. Added: pypy/trunk/src/pypy/annotation/__init__.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/annotation/__init__.py Wed Nov 26 15:18:24 2003 @@ -0,0 +1 @@ +# Copied: pypy/trunk/src/pypy/annotation/annset.py (from rev 2265, pypy/trunk/src/pypy/translator/annheap.py) ============================================================================== --- pypy/trunk/src/pypy/translator/annheap.py (original) +++ pypy/trunk/src/pypy/annotation/annset.py Wed Nov 26 15:18:24 2003 @@ -1,15 +1,32 @@ from __future__ import generators import types -from annotation import Annotation, XCell, XConstant, nothingyet +from model import Annotation, SomeValue, blackholevalue - -class AnnotationHeap: - """An annotation heap is a (large) family of Annotations.""" +class AnnotationSet: + """An annotation set is a (large) family of Annotations.""" # XXX STORED AS A PLAIN LIST, THE COMPLEXITY IS PLAINLY WRONG def __init__(self, annlist=[]): self.annlist = list(annlist) # List of annotations + self._shared = {} + + def getsharelist(self, someval): + return self._shared.get(someval, [someval]) + + def setshared(self, someval1, someval2): + list1 = self.getsharelist(someval1) + list2 = self.getsharelist(someval2) + newlist = list1 + list2 + for someval in newlist: + self._shared[someval] = newlist + + def isshared(self, someval1, someval2): + return (self._shared.get(someval1, someval1) is + self._shared.get(someval2, someval2)) + + def tempid(self, someval): + return id(self.getsharelist(someval)[0]) def dump(self): # debugging for ann in self.enumerate(): @@ -21,6 +38,47 @@ __iter__ = enumerate + def query(self, query, *querylist): + # slightly limited implementation for ease of coding :-) + results = [] + for match in self._getmatches(query): + # does the returned match also agree with the other queries? + for queryann in querylist: + boundquery = queryann.copy(renameargs={Ellipsis: match}) + if not self.contains(boundquery): + break + else: + results.append(match) + return results + + def contains(self, checkann): + for ann in self.annlist: + if ann.predicate == checkann.predicate: + for a1, a2 in zip(ann.args, checkann.args): + if not self.isshared(a1, a2): + break + else: + return True + return False + + def _getmatches(self, queryann): + assert queryann.args.count(Ellipsis) == 1, ( + "sorry, the algorithm is a bit too naive for this case") + queryarg = queryann.args.index(Ellipsis) + testindices = range(queryann.predicate.arity) + del testindices[queryarg] + for ann in self.annlist: + if ann.predicate == queryann.predicate: + for i in testindices: + if not self.isshared(ann.args[i], queryann.args[i]): + break + else: + yield ann.args[queryarg] + +# do you have an intersection algo somewherE? no mmmh we need to use +# self._shared too... + +''' def simplify(self, kill=[]): """Kill annotations in the list, and recursively all the annotations that depend on them, and simplify the resulting heap to remove @@ -108,19 +166,7 @@ # apply changes self.simplify(kill=deleting) return newcell - - -def rename(ann, oldcell, newcell): - "Make a copy of 'ann' in which 'oldcell' has been replaced by 'newcell'." - args = [] - for a in ann.args: - if a == oldcell: - a = newcell - args.append(a) - a = ann.result - if a == oldcell: - a = newcell - return Annotation(ann.opname, args, a) +''' class Transaction: Added: pypy/trunk/src/pypy/annotation/model.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/annotation/model.py Wed Nov 26 15:18:24 2003 @@ -0,0 +1,60 @@ + +class SomeValue: + pass + +# a conventional value for representing 'all Annotations match this one' +blackholevalue = SomeValue() + +def debugname(someval, name=None, _seen = {id(blackholevalue): 'blackholevalue'}): + """ return a simple name for a SomeValue. """ + try: + return _seen[id(someval)] + except KeyError: + if name is None: + name = "X%d" % len(seen) + _seen[id(someval)] = name + return name + +class Predicate: + def __init__(self, name, arity): + self.name = name + self.arity = arity + def __getitem__(self, args): + if self.arity == 1: + args = (args,) + return Annotation(self, *args) + +class ann: + add = Predicate('add', 3) + snuff = Predicate('snuff', 2) # for testing, to remove :-) + +class Annotation: + """An Annotation asserts something about SomeValues. + It is a Predicate applied to some arguments. """ + + def __init__(self, predicate, *args): + self.predicate = predicate # the operation or predicate + self.args = list(args) # list of SomeValues + assert len(args) == predicate.arity + # note that for predicates that are simple operations like + # op.add, the result is stored as the last argument. + for someval in args: + assert someval is Ellipsis or isinstance(someval, SomeValue) # bug catcher + + def copy(self, renameargs={}): + args = [renameargs.get(arg, arg) for arg in self.args] + return Annotation(self.predicate, *args) + + def __eq__(self, other): + return (self.__class__ is other.__class__ and + self.predicate == other.predicate and + self.args == other.args) + + def __ne__(self, other): + return not (self == other) + + def __repr__(self): + return "Annotation(%s, %s)" % ( + self.predicate, ", ".join(map(repr, self.args))) + + Modified: pypy/trunk/src/pypy/annotation/test/test_annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/test/test_annset.py (original) +++ pypy/trunk/src/pypy/annotation/test/test_annset.py Wed Nov 26 15:18:24 2003 @@ -2,13 +2,9 @@ import autopath from pypy.tool import test -from pypy.annotation.model import Annotation, SomeValue +from pypy.annotation.model import ann, SomeValue +from pypy.annotation.annset import AnnotationSet -# to avoid quoting of strings -class _op: - def __getattr__(self, name): - return name -op = _op() class TestAnnotationSet(test.IntTestCase): @@ -35,30 +31,30 @@ id1 = a.tempid(c1) id2 = a.tempid(c2) self.assertNotEquals(id1, id2) - + def test_query_one_annotation_arg(self): c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() - lst = [Annotation(op.add, c1, c3, c2)] + lst = [ann.add[c1, c3, c2]] a = AnnotationSet(lst) - c = a.query(Annotation(op.add, c1, c3, QUERYARG)) + c = a.query(ann.add[c1, c3, ...]) self.assertEquals(c, [c2]) - c = a.query(Annotation(op.add, c1, QUERYARG, c2)) + c = a.query(ann.add[c1, ..., c2]) self.assertEquals(c, [c3]) - c = a.query(Annotation(op.add, QUERYARG, c3, c2)) + c = a.query(ann.add[..., c3, c2]) self.assertEquals(c, [c1]) - c = a.query(Annotation(op.add, QUERYARG, c1, c2)) + c = a.query(ann.add[..., c1, c2]) self.assertEquals(c, []) def test_query_multiple_annotations(self): c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() lst = [ - Annotation(op.add, c1, c3, c2), - Annotation(op.something, c2, c3) + ann.add[c1, c3, c2], + ann.snuff[c2, c3], ] a = AnnotationSet(lst) - c = a.query(Annotation(op.add, c1, c3, QUERYARG), - Annotation(op.something, QUERYARG, c3)) + c = a.query(ann.add[c1, c3, ...], + ann.snuff[..., c3]) self.assertEquals(c, [c2]) if __name__ == '__main__': From arigo at codespeak.net Wed Nov 26 18:17:13 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 26 Nov 2003 18:17:13 +0100 (MET) Subject: [pypy-svn] rev 2268 - pypy/trunk/src/pypy/annotation Message-ID: <20031126171713.704F45B9A8@thoth.codespeak.net> Author: arigo Date: Wed Nov 26 18:17:12 2003 New Revision: 2268 Modified: pypy/trunk/src/pypy/annotation/annset.py Log: - Reorganization to separate again AnnotationSet from Transaction, which is now renamed to Recorder, which is closer to its purpose. Modified: pypy/trunk/src/pypy/annotation/annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/annset.py (original) +++ pypy/trunk/src/pypy/annotation/annset.py Wed Nov 26 18:17:12 2003 @@ -38,45 +38,70 @@ __iter__ = enumerate - def query(self, query, *querylist): + def query(self, *querylist): + return [match for depends, match in self.getmatches(*querylist)] + + def getmatches(self, query, *querylist): # slightly limited implementation for ease of coding :-) - results = [] - for match in self._getmatches(query): + assert query.args.count(Ellipsis) == 1, ( + "sorry, the algorithm is a bit too naive for this case") + queryarg = query.args.index(Ellipsis) + for ann in self._annmatch(query): # does the returned match also agree with the other queries? + match = ann.args[queryarg] + depends = [ann] for queryann in querylist: - boundquery = queryann.copy(renameargs={Ellipsis: match}) - if not self.contains(boundquery): + boundquery = queryann.copy(renameargs={Ellipsis: match}) + ann = self.findfirst(boundquery) + if ann is None: break + depends.append(ann) else: - results.append(match) - return results - - def contains(self, checkann): - for ann in self.annlist: - if ann.predicate == checkann.predicate: - for a1, a2 in zip(ann.args, checkann.args): - if not self.isshared(a1, a2): - break - else: - return True - return False + yield depends, match - def _getmatches(self, queryann): - assert queryann.args.count(Ellipsis) == 1, ( - "sorry, the algorithm is a bit too naive for this case") - queryarg = queryann.args.index(Ellipsis) - testindices = range(queryann.predicate.arity) - del testindices[queryarg] + def _annmatch(self, queryann): + testindices = [i for i in range(queryann.predicate.arity) + if queryann.args[i] is not Ellipsis] for ann in self.annlist: if ann.predicate == queryann.predicate: for i in testindices: if not self.isshared(ann.args[i], queryann.args[i]): break else: - yield ann.args[queryarg] + yield ann -# do you have an intersection algo somewherE? no mmmh we need to use -# self._shared too... + def findfirst(self, checkann): + """ return the first matching annotation.""" + # note that we are usually not interested in multiple matching + # annotations; e.g. killing an annotation will take care + # that all matching annotations are removed, and thus also + # all dependencies listed on any of the duplicate annotation. + for ann in self._annmatch(checkann): + return ann # :-) + else: + return None + + +class Recorder: + """A recorder contains methods to look for annotations in the + AnnotationSet and create new annotations accordingly. Each + Recorder instance records which Annotations were needed, which + allows dependencies to be tracked.""" + + def __init__(self, annset): + self.annset = annset + self.using_annotations = [] # annotations that we have used + + def using(self, *annlist): + """Mark all 'ann' in 'annlist' as used in this transaction.""" + self.using_annotations += annlist + + def query(self, *querylist): + results = [] + for depends, match in self.annset.getmatches(*querylist): + self.using(*depends) + results.append(match) + return results ''' def simplify(self, kill=[]): @@ -169,7 +194,7 @@ ''' -class Transaction: +class XXXTransaction: """A transaction contains methods to look for annotations in the AnnotationHeap and create new annotations accordingly. Each Transaction instance records which Annotations were needed, which From hpk at codespeak.net Wed Nov 26 19:27:32 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Wed, 26 Nov 2003 19:27:32 +0100 (MET) Subject: [pypy-svn] rev 2269 - in pypy/trunk/src/pypy/annotation: . test Message-ID: <20031126182732.C563E5B9A8@thoth.codespeak.net> Author: hpk Date: Wed Nov 26 19:27:30 2003 New Revision: 2269 Modified: pypy/trunk/src/pypy/annotation/README.txt pypy/trunk/src/pypy/annotation/annset.py pypy/trunk/src/pypy/annotation/model.py pypy/trunk/src/pypy/annotation/test/test_annset.py Log: added some more meat to AnnotationSet, mainly tests and implementation for the new 'recording' facility. See the new para in annotation/README.txt (armin, holger) Modified: pypy/trunk/src/pypy/annotation/README.txt ============================================================================== --- pypy/trunk/src/pypy/annotation/README.txt (original) +++ pypy/trunk/src/pypy/annotation/README.txt Wed Nov 26 19:27:30 2003 @@ -37,3 +37,11 @@ annotations, more possibilities. Annotations are stored in a global list, which is an AnnotationSet instance. AnnotationSet provides (via Transactions) methods to query, add and kill annotations. It also manages "sharing": two different SomeValue's can be later found to be identical (in the Python sense of "is"), and the AnnotationSet can be taught about this. + +You can't directly add annotations to an AnnotationSet. Adding an +annotation is considered to be dependent on previous annotations. +Thus you invoke annset.record(func), and your function 'func' will +be invoked with a 'Recorder' instance: you perform queries with it +and when you add/set a new annotation the recorder will remember +the dependency of the previous (queried) annotation towards the +new annotation. Modified: pypy/trunk/src/pypy/annotation/annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/annset.py (original) +++ pypy/trunk/src/pypy/annotation/annset.py Wed Nov 26 19:27:30 2003 @@ -10,6 +10,7 @@ def __init__(self, annlist=[]): self.annlist = list(annlist) # List of annotations self._shared = {} + self.forward_deps = {} def getsharelist(self, someval): return self._shared.get(someval, [someval]) @@ -28,6 +29,18 @@ def tempid(self, someval): return id(self.getsharelist(someval)[0]) + def annequal(self, ann1, ann2): + if ann1.predicate != ann2.predicate: + return False + for a1, a2 in zip(ann1.args, ann2.args): + if not self.isshared(a1, a2): + return False + return True + + def temporarykey(self, ann): + """ a temporary hashable representation of an annotation """ + return (ann.predicate, tuple([self.tempid(arg) for arg in ann.args])) + def dump(self): # debugging for ann in self.enumerate(): print ann @@ -81,65 +94,83 @@ else: return None + def record(self, recfunc, *args): + rec = Recorder(self) + return recfunc(rec, *args) -class Recorder: - """A recorder contains methods to look for annotations in the - AnnotationSet and create new annotations accordingly. Each - Recorder instance records which Annotations were needed, which - allows dependencies to be tracked.""" - - def __init__(self, annset): - self.annset = annset - self.using_annotations = [] # annotations that we have used - - def using(self, *annlist): - """Mark all 'ann' in 'annlist' as used in this transaction.""" - self.using_annotations += annlist - - def query(self, *querylist): - results = [] - for depends, match in self.annset.getmatches(*querylist): - self.using(*depends) - results.append(match) - return results + def kill(self, *annlist): + self.simplify(kill=annlist) -''' def simplify(self, kill=[]): """Kill annotations in the list, and recursively all the annotations - that depend on them, and simplify the resulting heap to remove + that depend on them, and simplify the resulting list to remove duplicates.""" # temporarykey() returns a tuple with all the information about # the annotation; equal temporarykey() means equal annotations. - # Such keys are temporary because making new XCells shared can + # Such keys are temporary because making SomeValues shared can # change the temporarykey(), but this doesn't occur during # one call to simplify(). allkeys = {} # map temporarykeys to Annotation instances for ann in self.annlist: - key = ann.temporarykey() + key = self.temporarykey(ann) if key in allkeys: # duplicate? previous = allkeys[key] - previous.forward_deps += ann.forward_deps # merge + if ann in self.forward_deps: + deps = self.forward_deps.setdefault(previous, []) + deps += self.forward_deps[ann] # merge else: allkeys[key] = ann killkeys = {} # set of temporarykeys of annotations to remove for ann in kill: - killkeys[ann.temporarykey()] = True - + killkeys[self.temporarykey(ann)] = True + pending = killkeys.keys() for key in pending: if key in allkeys: ann = allkeys[key] del allkeys[key] # remove annotations from the dict - for dep in ann.forward_deps: # propagate dependencies - depkey = dep.temporarykey() - if depkey not in killkeys: - killkeys[depkey] = True - pending.append(depkey) + if ann in self.forward_deps: + for dep in self.forward_deps[ann]: # propagate dependencies + depkey = self.temporarykey(dep) + if depkey not in killkeys: + killkeys[depkey] = True + pending.append(depkey) + del self.forward_deps[ann] self.annlist = allkeys.values() + +class Recorder: + """A recorder contains methods to look for annotations in the + AnnotationSet and create new annotations accordingly. Each + Recorder instance records which Annotations were needed, which + allows dependencies to be tracked.""" + + def __init__(self, annset): + self.annset = annset + self.using_annotations = [] # annotations that we have used + + def using(self, *annlist): + """Mark all 'ann' in 'annlist' as used in this transaction.""" + self.using_annotations += annlist + + def query(self, *querylist): + results = [] + for depends, match in self.annset.getmatches(*querylist): + self.using(*depends) + results.append(match) + return results + + def set(self, ann): + """Insert the annotation into the AnnotationSet, recording dependency + from all previous queries done on this Recorder instance.""" + self.annset.annlist.append(ann) + for previous_ann in self.using_annotations: + deps = self.annset.forward_deps.setdefault(previous_ann, []) + deps.append(ann) +''' def merge(self, oldcell, newcell): """Update the heap to account for the merging of oldcell and newcell. Return the merged cell.""" Modified: pypy/trunk/src/pypy/annotation/model.py ============================================================================== --- pypy/trunk/src/pypy/annotation/model.py (original) +++ pypy/trunk/src/pypy/annotation/model.py Wed Nov 26 19:27:30 2003 @@ -23,6 +23,8 @@ if self.arity == 1: args = (args,) return Annotation(self, *args) + def __str__(self): + return self.name class ann: add = Predicate('add', 3) @@ -45,14 +47,6 @@ args = [renameargs.get(arg, arg) for arg in self.args] return Annotation(self.predicate, *args) - def __eq__(self, other): - return (self.__class__ is other.__class__ and - self.predicate == other.predicate and - self.args == other.args) - - def __ne__(self, other): - return not (self == other) - def __repr__(self): return "Annotation(%s, %s)" % ( self.predicate, ", ".join(map(repr, self.args))) Modified: pypy/trunk/src/pypy/annotation/test/test_annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/test/test_annset.py (original) +++ pypy/trunk/src/pypy/annotation/test/test_annset.py Wed Nov 26 19:27:30 2003 @@ -31,6 +31,13 @@ id1 = a.tempid(c1) id2 = a.tempid(c2) self.assertNotEquals(id1, id2) + + def test_annequal(self): + c1,c2,c3,c4 = SomeValue(), SomeValue(), SomeValue(), SomeValue() + a = AnnotationSet() + a.setshared(c1,c4) + self.assert_(a.annequal(ann.add[c1,c2,c3], + ann.add[c4,c2,c3])) def test_query_one_annotation_arg(self): c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() @@ -57,5 +64,36 @@ ann.snuff[..., c3]) self.assertEquals(c, [c2]) +class TestRecording(test.IntTestCase): + + def assertSameSet(self, annset, a, b): + a = [annset.temporarykey(a1) for a1 in a] + b = [annset.temporarykey(b1) for b1 in b] + # try to reorder a to match b, without failing if the lists + # are different -- this will be checked by assertEquals() + for i in range(len(b)): + try: + j = i + a[i:].index(b[i]) + except ValueError: + pass + else: + a[i], a[j] = a[j], a[i] + self.assertEquals(a, b) + + def test_simple(self): + c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() + lst = [ + ann.add[c1, c3, c2], + ] + a = AnnotationSet(lst) + def f(rec): + if rec.query(ann.add[c1, c3, ...]): + rec.set(ann.snuff[c1, c3]) + a.record(f) + self.assertSameSet(a, a, lst + [ann.snuff[c1, c3]]) + + a.kill(lst[0]) + self.assertSameSet(a, a, []) + if __name__ == '__main__': test.main() From arigo at codespeak.net Wed Nov 26 23:47:27 2003 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 26 Nov 2003 23:47:27 +0100 (MET) Subject: [pypy-svn] rev 2270 - in pypy/trunk/src/pypy/annotation: . test Message-ID: <20031126224727.8C01A5B9A8@thoth.codespeak.net> Author: arigo Date: Wed Nov 26 23:47:26 2003 New Revision: 2270 Modified: pypy/trunk/src/pypy/annotation/annset.py pypy/trunk/src/pypy/annotation/model.py pypy/trunk/src/pypy/annotation/test/test_annset.py Log: intermediate checkin - added a constant-predicate - added get_type/check_type methods to annset.py - added intvalue, strvalue, etc. but don't hold your breath, next check-in will probably remove them anyway :-) (arminl holgger on laggy screen ssseeeession ) Modified: pypy/trunk/src/pypy/annotation/annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/annset.py (original) +++ pypy/trunk/src/pypy/annotation/annset.py Wed Nov 26 23:47:26 2003 @@ -1,13 +1,14 @@ from __future__ import generators import types -from model import Annotation, SomeValue, blackholevalue +from model import Annotation, SomeValue, ann +from model import immutable_types, blackholevalue, basicannotations class AnnotationSet: """An annotation set is a (large) family of Annotations.""" # XXX STORED AS A PLAIN LIST, THE COMPLEXITY IS PLAINLY WRONG - def __init__(self, annlist=[]): + def __init__(self, annlist=basicannotations): self.annlist = list(annlist) # List of annotations self._shared = {} self.forward_deps = {} @@ -170,6 +171,18 @@ for previous_ann in self.using_annotations: deps = self.annset.forward_deps.setdefault(previous_ann, []) deps.append(ann) + + def check_type(self, someval, checktype): + return self.query(ann.type[someval, ...], + ann.constant(checktype)[...]) + + def set_type(self, someval, knowntype): + typeval = SomeValue() + self.set(ann.type[someval, typeval]) + self.set(ann.constant(knowntype)[typeval]) + if knowntype in immutable_types: + self.set(ann.immutable[someval]) + ''' def merge(self, oldcell, newcell): """Update the heap to account for the merging of oldcell and newcell. Modified: pypy/trunk/src/pypy/annotation/model.py ============================================================================== --- pypy/trunk/src/pypy/annotation/model.py (original) +++ pypy/trunk/src/pypy/annotation/model.py Wed Nov 26 23:47:26 2003 @@ -1,17 +1,19 @@ +import types class SomeValue: pass -# a conventional value for representing 'all Annotations match this one' -blackholevalue = SomeValue() - -def debugname(someval, name=None, _seen = {id(blackholevalue): 'blackholevalue'}): +def debugname(someval, _seen = {}): """ return a simple name for a SomeValue. """ try: return _seen[id(someval)] except KeyError: - if name is None: - name = "X%d" % len(seen) + if not _seen: + for name, value in globals().items(): + if isinstance(value, SomeValue): + _seen[id(value)] = name + return debugname(someval) + name = "X%d" % len(seen) _seen[id(someval)] = name return name @@ -26,9 +28,23 @@ def __str__(self): return self.name +class ConstPredicate(Predicate): + def __init__(self, value): + Predicate.__init__(self, 'const%s' % value, 1) + self.value = value + def __eq__(self, other): + return self.__class__ is other.__class__ and self.value == other.value + def __ne__(self, other): + return not (self == other) + def __hash__(self): + return hash(self.value) + class ann: add = Predicate('add', 3) snuff = Predicate('snuff', 2) # for testing, to remove :-) + constant = ConstPredicate + type = Predicate('type', 2) + immutable = Predicate('immutable', 1) class Annotation: """An Annotation asserts something about SomeValues. @@ -52,3 +68,24 @@ self.predicate, ", ".join(map(repr, self.args))) +immutable_types = { + int: 'int', + long: 'long', + tuple: 'tuple', + str: 'str', + bool: 'bool', + types.FunctionType: 'function', + } + +# a conventional value for representing 'all Annotations match this one' +blackholevalue = SomeValue() + +# a few values representing 'any value of the given type' +# the following loops creates intvalue, strvalue, etc. +basicannotations = [] +for _type, _name in immutable_types.items(): + _val = globals()['%svalue' % _name] = SomeValue() + _tval = SomeValue() + basicannotations.append(ann.type[_val, _tval]) + basicannotations.append(ann.constant(_type)[_tval]) + basicannotations.append(ann.immutable[_val]) Modified: pypy/trunk/src/pypy/annotation/test/test_annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/test/test_annset.py (original) +++ pypy/trunk/src/pypy/annotation/test/test_annset.py Wed Nov 26 23:47:26 2003 @@ -64,8 +64,27 @@ ann.snuff[..., c3]) self.assertEquals(c, [c2]) + def test_constant(self): + c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() + lst = [ + ann.constant(42)[c1], + ] + a = AnnotationSet(lst) + c = a.query(ann.constant(42)[...]) + self.assertEquals(c, [c1]) + +c1,c2,c3,c4 = SomeValue(), SomeValue(), SomeValue(), SomeValue() + class TestRecording(test.IntTestCase): + def setUp(self): + self.lst = [ + ann.add[c1, c3, c2], + ann.type[c1, c4], + ann.constant(int)[c4], + ] + self.annset = AnnotationSet(self.lst) + def assertSameSet(self, annset, a, b): a = [annset.temporarykey(a1) for a1 in a] b = [annset.temporarykey(b1) for b1 in b] @@ -81,19 +100,24 @@ self.assertEquals(a, b) def test_simple(self): - c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() - lst = [ - ann.add[c1, c3, c2], - ] - a = AnnotationSet(lst) + a = self.annset def f(rec): if rec.query(ann.add[c1, c3, ...]): rec.set(ann.snuff[c1, c3]) a.record(f) - self.assertSameSet(a, a, lst + [ann.snuff[c1, c3]]) + self.assertSameSet(a, a, self.lst + [ann.snuff[c1, c3]]) + + a.kill(self.lst[0]) + self.assertSameSet(a, a, self.lst[1:]) - a.kill(lst[0]) - self.assertSameSet(a, a, []) + def test_type(self): + a = self.annset + def f(rec): + if rec.check_type(c1, int): + rec.set_type(c2, str) + a.record(f) + self.assert_(a.query(ann.type[c2, ...], + ann.constant(str)[...])) if __name__ == '__main__': test.main() From hpk at codespeak.net Thu Nov 27 16:31:38 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Thu, 27 Nov 2003 16:31:38 +0100 (MET) Subject: [pypy-svn] rev 2271 - pypy/trunk/src/pypy/annotation Message-ID: <20031127153138.73A235AB9F@thoth.codespeak.net> Author: hpk Date: Thu Nov 27 16:31:37 2003 New Revision: 2271 Modified: pypy/trunk/src/pypy/annotation/annset.py pypy/trunk/src/pypy/annotation/model.py Log: a couple of small enhancements/bug fixes. also python -i annset.py allows to play around with annotations Modified: pypy/trunk/src/pypy/annotation/annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/annset.py (original) +++ pypy/trunk/src/pypy/annotation/annset.py Thu Nov 27 16:31:37 2003 @@ -325,3 +325,8 @@ bool: True, types.FunctionType: True, } + +if __name__ == '__main__': + val1, val2, val3 = SomeValue(), SomeValue(), SomeValue() + annset = AnnotationSet() + Modified: pypy/trunk/src/pypy/annotation/model.py ============================================================================== --- pypy/trunk/src/pypy/annotation/model.py (original) +++ pypy/trunk/src/pypy/annotation/model.py Thu Nov 27 16:31:37 2003 @@ -13,20 +13,20 @@ if isinstance(value, SomeValue): _seen[id(value)] = name return debugname(someval) - name = "X%d" % len(seen) + name = "V%d" % len(_seen) _seen[id(someval)] = name return name class Predicate: - def __init__(self, name, arity): - self.name = name + def __init__(self, debugname, arity): + self.debugname = debugname self.arity = arity def __getitem__(self, args): if self.arity == 1: args = (args,) return Annotation(self, *args) def __str__(self): - return self.name + return self.debugname class ConstPredicate(Predicate): def __init__(self, value): @@ -65,7 +65,7 @@ def __repr__(self): return "Annotation(%s, %s)" % ( - self.predicate, ", ".join(map(repr, self.args))) + self.predicate, ", ".join(map(debugname, self.args))) immutable_types = { From hpk at codespeak.net Thu Nov 27 17:27:22 2003 From: hpk at codespeak.net (hpk at codespeak.net) Date: Thu, 27 Nov 2003 17:27:22 +0100 (MET) Subject: [pypy-svn] rev 2272 - in pypy/trunk/src/pypy/annotation: . test Message-ID: <20031127162722.D4B0F5ACB1@thoth.codespeak.net> Author: hpk Date: Thu Nov 27 17:27:21 2003 New Revision: 2272 Modified: pypy/trunk/src/pypy/annotation/annset.py pypy/trunk/src/pypy/annotation/model.py pypy/trunk/src/pypy/annotation/test/test_annset.py Log: - got rid of the 'snuff' predicate - some more more little renamings for the fun of it (and while i am waiting on armin :-) Modified: pypy/trunk/src/pypy/annotation/annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/annset.py (original) +++ pypy/trunk/src/pypy/annotation/annset.py Thu Nov 27 17:27:21 2003 @@ -1,6 +1,6 @@ from __future__ import generators import types -from model import Annotation, SomeValue, ann +from model import Annotation, SomeValue, ANN from model import immutable_types, blackholevalue, basicannotations class AnnotationSet: @@ -53,14 +53,18 @@ __iter__ = enumerate def query(self, *querylist): - return [match for depends, match in self.getmatches(*querylist)] + return [matchvalue for matchanns, matchvalue in self.match(*querylist)] + + def match(self, query, *querylist): + """ yield (matchanns, matchvalue) tuples with 'matchanns' + beeing a list of matching annotations and 'matchvalue' beeing + the queried value. """ - def getmatches(self, query, *querylist): # slightly limited implementation for ease of coding :-) assert query.args.count(Ellipsis) == 1, ( "sorry, the algorithm is a bit too naive for this case") queryarg = query.args.index(Ellipsis) - for ann in self._annmatch(query): + for ann in self._annmatches(query): # does the returned match also agree with the other queries? match = ann.args[queryarg] depends = [ann] @@ -73,9 +77,10 @@ else: yield depends, match - def _annmatch(self, queryann): + def _annmatches(self, queryann): + """ yield annotations matching the given queryannotation. """ testindices = [i for i in range(queryann.predicate.arity) - if queryann.args[i] is not Ellipsis] + if queryann.args[i] is not Ellipsis] for ann in self.annlist: if ann.predicate == queryann.predicate: for i in testindices: @@ -90,7 +95,7 @@ # annotations; e.g. killing an annotation will take care # that all matching annotations are removed, and thus also # all dependencies listed on any of the duplicate annotation. - for ann in self._annmatch(checkann): + for ann in self._annmatches(checkann): return ann # :-) else: return None @@ -159,9 +164,9 @@ def query(self, *querylist): results = [] - for depends, match in self.annset.getmatches(*querylist): - self.using(*depends) - results.append(match) + for matchanns, matchvalue in self.annset.match(*querylist): + self.using(*matchanns) + results.append(matchvalue) return results def set(self, ann): @@ -173,15 +178,15 @@ deps.append(ann) def check_type(self, someval, checktype): - return self.query(ann.type[someval, ...], - ann.constant(checktype)[...]) + return self.query(ANN.type[someval, ...], + ANN.constant(checktype)[...]) def set_type(self, someval, knowntype): typeval = SomeValue() - self.set(ann.type[someval, typeval]) - self.set(ann.constant(knowntype)[typeval]) + self.set(ANN.type[someval, typeval]) + self.set(ANN.constant(knowntype)[typeval]) if knowntype in immutable_types: - self.set(ann.immutable[someval]) + self.set(ANN.immutable[someval]) ''' def merge(self, oldcell, newcell): @@ -235,9 +240,8 @@ # apply changes self.simplify(kill=deleting) return newcell -''' - +''' class XXXTransaction: """A transaction contains methods to look for annotations in the AnnotationHeap and create new annotations accordingly. Each Modified: pypy/trunk/src/pypy/annotation/model.py ============================================================================== --- pypy/trunk/src/pypy/annotation/model.py (original) +++ pypy/trunk/src/pypy/annotation/model.py Thu Nov 27 17:27:21 2003 @@ -3,20 +3,6 @@ class SomeValue: pass -def debugname(someval, _seen = {}): - """ return a simple name for a SomeValue. """ - try: - return _seen[id(someval)] - except KeyError: - if not _seen: - for name, value in globals().items(): - if isinstance(value, SomeValue): - _seen[id(value)] = name - return debugname(someval) - name = "V%d" % len(_seen) - _seen[id(someval)] = name - return name - class Predicate: def __init__(self, debugname, arity): self.debugname = debugname @@ -39,9 +25,8 @@ def __hash__(self): return hash(self.value) -class ann: +class ANN: add = Predicate('add', 3) - snuff = Predicate('snuff', 2) # for testing, to remove :-) constant = ConstPredicate type = Predicate('type', 2) immutable = Predicate('immutable', 1) @@ -67,6 +52,19 @@ return "Annotation(%s, %s)" % ( self.predicate, ", ".join(map(debugname, self.args))) +def debugname(someval, _seen = {}): + """ return a simple name for a SomeValue. """ + try: + return _seen[id(someval)] + except KeyError: + if not _seen: + for name, value in globals().items(): + if isinstance(value, SomeValue): + _seen[id(value)] = name + return debugname(someval) + name = "V%d" % len(_seen) + _seen[id(someval)] = name + return name immutable_types = { int: 'int', @@ -86,6 +84,6 @@ for _type, _name in immutable_types.items(): _val = globals()['%svalue' % _name] = SomeValue() _tval = SomeValue() - basicannotations.append(ann.type[_val, _tval]) - basicannotations.append(ann.constant(_type)[_tval]) - basicannotations.append(ann.immutable[_val]) + basicannotations.append(ANN.type[_val, _tval]) + basicannotations.append(ANN.constant(_type)[_tval]) + basicannotations.append(ANN.immutable[_val]) Modified: pypy/trunk/src/pypy/annotation/test/test_annset.py ============================================================================== --- pypy/trunk/src/pypy/annotation/test/test_annset.py (original) +++ pypy/trunk/src/pypy/annotation/test/test_annset.py Thu Nov 27 17:27:21 2003 @@ -2,7 +2,7 @@ import autopath from pypy.tool import test -from pypy.annotation.model import ann, SomeValue +from pypy.annotation.model import ANN, SomeValue from pypy.annotation.annset import AnnotationSet @@ -36,41 +36,41 @@ c1,c2,c3,c4 = SomeValue(), SomeValue(), SomeValue(), SomeValue() a = AnnotationSet() a.setshared(c1,c4) - self.assert_(a.annequal(ann.add[c1,c2,c3], - ann.add[c4,c2,c3])) + self.assert_(a.annequal(ANN.add[c1,c2,c3], + ANN.add[c4,c2,c3])) def test_query_one_annotation_arg(self): c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() - lst = [ann.add[c1, c3, c2]] + lst = [ANN.add[c1, c3, c2]] a = AnnotationSet(lst) - c = a.query(ann.add[c1, c3, ...]) + c = a.query(ANN.add[c1, c3, ...]) self.assertEquals(c, [c2]) - c = a.query(ann.add[c1, ..., c2]) + c = a.query(ANN.add[c1, ..., c2]) self.assertEquals(c, [c3]) - c = a.query(ann.add[..., c3, c2]) + c = a.query(ANN.add[..., c3, c2]) self.assertEquals(c, [c1]) - c = a.query(ann.add[..., c1, c2]) + c = a.query(ANN.add[..., c1, c2]) self.assertEquals(c, []) def test_query_multiple_annotations(self): c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() lst = [ - ann.add[c1, c3, c2], - ann.snuff[c2, c3], + ANN.add[c1, c3, c2], + ANN.type[c2, c3], ] a = AnnotationSet(lst) - c = a.query(ann.add[c1, c3, ...], - ann.snuff[..., c3]) + c = a.query(ANN.add[c1, c3, ...], + ANN.type[..., c3]) self.assertEquals(c, [c2]) def test_constant(self): c1,c2,c3 = SomeValue(), SomeValue(), SomeValue() lst = [ - ann.constant(42)[c1], + ANN.constant(42)[c1], ] a = AnnotationSet(lst) - c = a.query(ann.constant(42)[...]) + c = a.query(ANN.constant(42)[...]) self.assertEquals(c, [c1]) c1,c2,c3,c4 = SomeValue(), SomeValue(), SomeValue(), SomeValue() @@ -79,9 +79,9 @@ def setUp(self): self.lst = [ - ann.add[c1, c3, c2], - ann.type[c1, c4], - ann.constant(int)[c4], + ANN.add[c1, c3, c2], + ANN.type[c1, c4], + ANN.constant(int)[c4], ] self.annset = AnnotationSet(self.lst) @@ -102,10 +102,10 @@ def test_simple(self): a = self.annset def f(rec): - if rec.query(ann.add[c1, c3, ...]): - rec.set(ann.snuff[c1, c3]) + if rec.query(ANN.add[c1, c3, ...]): + rec.set(ANN.type[c1, c3]) a.record(f) - self.assertSameSet(a, a, self.lst + [ann.snuff[c1, c3]]) + self.assertSameSet(a, a, self.lst + [ANN.type[c1, c3]]) a.kill(self.lst[0]) self.assertSameSet(a, a, self.lst[1:]) @@ -116,8 +116,8 @@ if rec.check_type(c1, int): rec.set_type(c2, str) a.record(f) - self.assert_(a.query(ann.type[c2, ...], - ann.constant(str)[...])) + self.assert_(a.query(ANN.type[c2, ...], + ANN.constant(str)[...])) if __name__ == '__main__': test.main() From rocco at codespeak.net Sun Nov 30 18:53:13 2003 From: rocco at codespeak.net (rocco at codespeak.net) Date: Sun, 30 Nov 2003 18:53:13 +0100 (MET) Subject: [pypy-svn] rev 2280 - in pypy/trunk/src/pypy: annotation appspace module module/test Message-ID: <20031130175313.C6AA15BE97@thoth.codespeak.net> Author: rocco Date: Sun Nov 30 18:53:12 2003 New Revision: 2280 Added: pypy/trunk/src/pypy/appspace/copy_reg.py (contents, props changed) pypy/trunk/src/pypy/appspace/os.py (contents, props changed) pypy/trunk/src/pypy/module/_randommodule.py (contents, props changed) pypy/trunk/src/pypy/module/_sremodule.py (contents, props changed) pypy/trunk/src/pypy/module/cStringIOmodule.py (contents, props changed) pypy/trunk/src/pypy/module/itertoolsmodule.py (contents, props changed) pypy/trunk/src/pypy/module/mathmodule.py (contents, props changed) Modified: pypy/trunk/src/pypy/annotation/ (props changed) pypy/trunk/src/pypy/appspace/types.py pypy/trunk/src/pypy/module/__init__.py pypy/trunk/src/pypy/module/builtin.py pypy/trunk/src/pypy/module/sysmodule.py pypy/trunk/src/pypy/module/test/test_builtin.py pypy/trunk/src/pypy/module/test/test_sysmodule.py Log: Add functionality on the way towards getting CPython regression tests to run M module\sysmodule.py M module\test\test_sysmodule.py Add missing items M module\builtin.py M module\test\test_builtin.py Add missing item and fix execfile() so that tracebacks have proper filenames M module\__init__.py A module\itertoolsmodule.py A module\_sremodule.py A module\mathmodule.py A module\_randommodule.py A module\cStringIOmodule.py Add some missing builtin modules - (Note that these probably should get rewritten at some point when we get a functioning RPython implementation, but for now ... M appspace\types.py Add copyright notice, and some missing 2.3 items, while we're at it. A appspace\os.py dir() doesn't work, so we work around it for now Changed line 40: return [n for n in module.__dict__.keys() if n[0] != '_'] A appspace\copy_reg.py Remove irrelevant and harmful assertion Changed lines 18&19: ## if type(ob_type) is _ClassType: ## raise TypeError("copy_reg is not intended for use with classes") M annotation Ignore *.pyc files Added: pypy/trunk/src/pypy/appspace/copy_reg.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/appspace/copy_reg.py Sun Nov 30 18:53:12 2003 @@ -0,0 +1,192 @@ +"""Helper to provide extensibility for pickle/cPickle. + +!! This file has been copied practicaly verbatim from the CPython source. +!! See http://www.python.org/2.3.2/license.html for licensing info. + +This is only useful to add pickle support for extension types defined in +C, not for instances of user-defined classes. +""" + +from types import ClassType as _ClassType + +__all__ = ["pickle", "constructor", + "add_extension", "remove_extension", "clear_extension_cache"] + +dispatch_table = {} + +def pickle(ob_type, pickle_function, constructor_ob=None): +## if type(ob_type) is _ClassType: +## raise TypeError("copy_reg is not intended for use with classes") + + if not callable(pickle_function): + raise TypeError("reduction functions must be callable") + dispatch_table[ob_type] = pickle_function + + # The constructor_ob function is a vestige of safe for unpickling. + # There is no reason for the caller to pass it anymore. + if constructor_ob is not None: + constructor(constructor_ob) + +def constructor(object): + if not callable(object): + raise TypeError("constructors must be callable") + +# Example: provide pickling support for complex numbers. + +try: + complex +except NameError: + pass +else: + + def pickle_complex(c): + return complex, (c.real, c.imag) + + pickle(complex, pickle_complex, complex) + +# Support for pickling new-style objects + +def _reconstructor(cls, base, state): + if base is object: + obj = object.__new__(cls) + else: + obj = base.__new__(cls, state) + base.__init__(obj, state) + return obj + +_HEAPTYPE = 1<<9 + +# Python code for object.__reduce_ex__ for protocols 0 and 1 + +def _reduce_ex(self, proto): + assert proto < 2 + for base in self.__class__.__mro__: + if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE: + break + else: + base = object # not really reachable + if base is object: + state = None + else: + if base is self.__class__: + raise TypeError, "can't pickle %s objects" % base.__name__ + state = base(self) + args = (self.__class__, base, state) + try: + getstate = self.__getstate__ + except AttributeError: + if getattr(self, "__slots__", None): + raise TypeError("a class that defines __slots__ without " + "defining __getstate__ cannot be pickled") + try: + dict = self.__dict__ + except AttributeError: + dict = None + else: + dict = getstate() + if dict: + return _reconstructor, args, dict + else: + return _reconstructor, args + +# Helper for __reduce_ex__ protocol 2 + +def __newobj__(cls, *args): + return cls.__new__(cls, *args) + +def _slotnames(cls): + """Return a list of slot names for a given class. + + This needs to find slots defined by the class and its bases, so we + can't simply return the __slots__ attribute. We must walk down + the Method Resolution Order and concatenate the __slots__ of each + class found there. (This assumes classes don't modify their + __slots__ attribute to misrepresent their slots after the class is + defined.) + """ + + # Get the value from a cache in the class if possible + names = cls.__dict__.get("__slotnames__") + if names is not None: + return names + + # Not cached -- calculate the value + names = [] + if not hasattr(cls, "__slots__"): + # This class has no slots + pass + else: + # Slots found -- gather slot names from all base classes + for c in cls.__mro__: + if "__slots__" in c.__dict__: + names += [name for name in c.__dict__["__slots__"] + if name not in ("__dict__", "__weakref__")] + + # Cache the outcome in the class if at all possible + try: + cls.__slotnames__ = names + except: + pass # But don't die if we can't + + return names + +# A registry of extension codes. This is an ad-hoc compression +# mechanism. Whenever a global reference to , is about +# to be pickled, the (, ) tuple is looked up here to see +# if it is a registered extension code for it. Extension codes are +# universal, so that the meaning of a pickle does not depend on +# context. (There are also some codes reserved for local use that +# don't have this restriction.) Codes are positive ints; 0 is +# reserved. + +_extension_registry = {} # key -> code +_inverted_registry = {} # code -> key +_extension_cache = {} # code -> object +# Don't ever rebind those names: cPickle grabs a reference to them when +# it's initialized, and won't see a rebinding. + +def add_extension(module, name, code): + """Register an extension code.""" + code = int(code) + if not 1 <= code <= 0x7fffffff: + raise ValueError, "code out of range" + key = (module, name) + if (_extension_registry.get(key) == code and + _inverted_registry.get(code) == key): + return # Redundant registrations are benign + if key in _extension_registry: + raise ValueError("key %s is already registered with code %s" % + (key, _extension_registry[key])) + if code in _inverted_registry: + raise ValueError("code %s is already in use for key %s" % + (code, _inverted_registry[code])) + _extension_registry[key] = code + _inverted_registry[code] = key + +def remove_extension(module, name, code): + """Unregister an extension code. For testing only.""" + key = (module, name) + if (_extension_registry.get(key) != code or + _inverted_registry.get(code) != key): + raise ValueError("key %s is not registered with code %s" % + (key, code)) + del _extension_registry[key] + del _inverted_registry[code] + if code in _extension_cache: + del _extension_cache[code] + +def clear_extension_cache(): + _extension_cache.clear() + +# Standard extension code assignments + +# Reserved ranges + +# First Last Count Purpose +# 1 127 127 Reserved for Python standard library +# 128 191 64 Reserved for Zope +# 192 239 48 Reserved for 3rd parties +# 240 255 16 Reserved for private use (will never be assigned) +# 256 Inf Inf Reserved for future assignment + +# Extension codes are assigned by the Python Software Foundation. Added: pypy/trunk/src/pypy/appspace/os.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/appspace/os.py Sun Nov 30 18:53:12 2003 @@ -0,0 +1,659 @@ +r"""OS routines for Mac, DOS, NT, or Posix depending on what system we're on. + +!! This file has been copied practicaly verbatim from the CPython source. +!! See http://www.python.org/2.3.2/license.html for licensing info. + +This exports: + - all functions from posix, nt, os2, mac, or ce, e.g. unlink, stat, etc. + - os.path is one of the modules posixpath, ntpath, or macpath + - os.name is 'posix', 'nt', 'os2', 'mac', 'ce' or 'riscos' + - os.curdir is a string representing the current directory ('.' or ':') + - os.pardir is a string representing the parent directory ('..' or '::') + - os.sep is the (or a most common) pathname separator ('/' or ':' or '\\') + - os.extsep is the extension separator ('.' or '/') + - os.altsep is the alternate pathname separator (None or '/') + - os.pathsep is the component separator used in $PATH etc + - os.linesep is the line separator in text files ('\r' or '\n' or '\r\n') + - os.defpath is the default search path for executables + +Programs that import and use 'os' stand a better chance of being +portable between different platforms. Of course, they must then +only use functions that are defined by all platforms (e.g., unlink +and opendir), and leave all pathname manipulation to os.path +(e.g., split and join). +""" + +#' + +import sys + +_names = sys.builtin_module_names + +# Note: more names are added to __all__ later. +__all__ = ["altsep", "curdir", "pardir", "sep", "pathsep", "linesep", + "defpath", "name", "path"] + +def _get_exports_list(module): + try: + return list(module.__all__) + except AttributeError: + return [n for n in module.__dict__.keys() if n[0] != '_'] + +if 'posix' in _names: + name = 'posix' + linesep = '\n' + from posix import * + try: + from posix import _exit + except ImportError: + pass + import posixpath as path + + import posix + __all__.extend(_get_exports_list(posix)) + del posix + +elif 'nt' in _names: + name = 'nt' + linesep = '\r\n' + from nt import * + try: + from nt import _exit + except ImportError: + pass + import ntpath as path + + import nt + __all__.extend(_get_exports_list(nt)) + del nt + +elif 'os2' in _names: + name = 'os2' + linesep = '\r\n' + from os2 import * + try: + from os2 import _exit + except ImportError: + pass + if sys.version.find('EMX GCC') == -1: + import ntpath as path + else: + import os2emxpath as path + + import os2 + __all__.extend(_get_exports_list(os2)) + del os2 + +elif 'mac' in _names: + name = 'mac' + linesep = '\r' + from mac import * + try: + from mac import _exit + except ImportError: + pass + import macpath as path + + import mac + __all__.extend(_get_exports_list(mac)) + del mac + +elif 'ce' in _names: + name = 'ce' + linesep = '\r\n' + from ce import * + try: + from ce import _exit + except ImportError: + pass + # We can use the standard Windows path. + import ntpath as path + + import ce + __all__.extend(_get_exports_list(ce)) + del ce + +elif 'riscos' in _names: + name = 'riscos' + linesep = '\n' + from riscos import * + try: + from riscos import _exit + except ImportError: + pass + import riscospath as path + + import riscos + __all__.extend(_get_exports_list(riscos)) + del riscos + +else: + raise ImportError, 'no os specific module found' + +sys.modules['os.path'] = path +from os.path import curdir, pardir, sep, pathsep, defpath, extsep, altsep + +del _names + +#' + +# Super directory utilities. +# (Inspired by Eric Raymond; the doc strings are mostly his) + +def makedirs(name, mode=0777): + """makedirs(path [, mode=0777]) + + Super-mkdir; create a leaf directory and all intermediate ones. + Works like mkdir, except that any intermediate path segment (not + just the rightmost) will be created if it does not exist. This is + recursive. + + """ + head, tail = path.split(name) + if not tail: + head, tail = path.split(head) + if head and tail and not path.exists(head): + makedirs(head, mode) + mkdir(name, mode) + +def removedirs(name): + """removedirs(path) + + Super-rmdir; remove a leaf directory and empty all intermediate + ones. Works like rmdir except that, if the leaf directory is + successfully removed, directories corresponding to rightmost path + segments will be pruned away until either the whole path is + consumed or an error occurs. Errors during this latter phase are + ignored -- they generally mean that a directory was not empty. + + """ + rmdir(name) + head, tail = path.split(name) + if not tail: + head, tail = path.split(head) + while head and tail: + try: + rmdir(head) + except error: + break + head, tail = path.split(head) + +def renames(old, new): + """renames(old, new) + + Super-rename; create directories as necessary and delete any left + empty. Works like rename, except creation of any intermediate + directories needed to make the new pathname good is attempted + first. After the rename, directories corresponding to rightmost + path segments of the old name will be pruned way until either the + whole path is consumed or a nonempty directory is found. + + Note: this function can fail with the new directory structure made + if you lack permissions needed to unlink the leaf directory or + file. + + """ + head, tail = path.split(new) + if head and tail and not path.exists(head): + makedirs(head) + rename(old, new) + head, tail = path.split(old) + if head and tail: + try: + removedirs(head) + except error: + pass + +__all__.extend(["makedirs", "removedirs", "renames"]) + +def walk(top, topdown=True, onerror=None): + """Directory tree generator. + + For each directory in the directory tree rooted at top (including top + itself, but excluding '.' and '..'), yields a 3-tuple + + dirpath, dirnames, filenames + + dirpath is a string, the path to the directory. dirnames is a list of + the names of the subdirectories in dirpath (excluding '.' and '..'). + filenames is a list of the names of the non-directory files in dirpath. + Note that the names in the lists are just names, with no path components. + To get a full path (which begins with top) to a file or directory in + dirpath, do os.path.join(dirpath, name). + + If optional arg 'topdown' is true or not specified, the triple for a + directory is generated before the triples for any of its subdirectories + (directories are generated top down). If topdown is false, the triple + for a directory is generated after the triples for all of its + subdirectories (directories are generated bottom up). + + When topdown is true, the caller can modify the dirnames list in-place + (e.g., via del or slice assignment), and walk will only recurse into the + subdirectories whose names remain in dirnames; this can be used to prune + the search, or to impose a specific order of visiting. Modifying + dirnames when topdown is false is ineffective, since the directories in + dirnames have already been generated by the time dirnames itself is + generated. + + By default errors from the os.listdir() call are ignored. If + optional arg 'onerror' is specified, it should be a function; it + will be called with one argument, an os.error instance. It can + report the error to continue with the walk, or raise the exception + to abort the walk. Note that the filename is available as the + filename attribute of the exception object. + + Caution: if you pass a relative pathname for top, don't change the + current working directory between resumptions of walk. walk never + changes the current directory, and assumes that the client doesn't + either. + + Example: + + from os.path import join, getsize + for root, dirs, files in walk('python/Lib/email'): + print root, "consumes", + print sum([getsize(join(root, name)) for name in files]), + print "bytes in", len(files), "non-directory files" + if 'CVS' in dirs: + dirs.remove('CVS') # don't visit CVS directories + """ + + from os.path import join, isdir, islink + + # We may not have read permission for top, in which case we can't + # get a list of the files the directory contains. os.path.walk + # always suppressed the exception then, rather than blow up for a + # minor reason when (say) a thousand readable directories are still + # left to visit. That logic is copied here. + try: + # Note that listdir and error are globals in this module due + # to earlier import-*. + names = listdir(top) + except error, err: + if onerror is not None: + onerror(err) + return + + dirs, nondirs = [], [] + for name in names: + if isdir(join(top, name)): + dirs.append(name) + else: + nondirs.append(name) + + if topdown: + yield top, dirs, nondirs + for name in dirs: + path = join(top, name) + if not islink(path): + for x in walk(path, topdown, onerror): + yield x + if not topdown: + yield top, dirs, nondirs + +__all__.append("walk") + +# Make sure os.environ exists, at least +try: + environ +except NameError: + environ = {} + +def execl(file, *args): + """execl(file, *args) + + Execute the executable file with argument list args, replacing the + current process. """ + execv(file, args) + +def execle(file, *args): + """execle(file, *args, env) + + Execute the executable file with argument list args and + environment env, replacing the current process. """ + env = args[-1] + execve(file, args[:-1], env) + +def execlp(file, *args): + """execlp(file, *args) + + Execute the executable file (which is searched for along $PATH) + with argument list args, replacing the current process. """ + execvp(file, args) + +def execlpe(file, *args): + """execlpe(file, *args, env) + + Execute the executable file (which is searched for along $PATH) + with argument list args and environment env, replacing the current + process. """ + env = args[-1] + execvpe(file, args[:-1], env) + +def execvp(file, args): + """execp(file, args) + + Execute the executable file (which is searched for along $PATH) + with argument list args, replacing the current process. + args may be a list or tuple of strings. """ + _execvpe(file, args) + +def execvpe(file, args, env): + """execvpe(file, args, env) + + Execute the executable file (which is searched for along $PATH) + with argument list args and environment env , replacing the + current process. + args may be a list or tuple of strings. """ + _execvpe(file, args, env) + +__all__.extend(["execl","execle","execlp","execlpe","execvp","execvpe"]) + +def _execvpe(file, args, env=None): + from errno import ENOENT, ENOTDIR + + if env is not None: + func = execve + argrest = (args, env) + else: + func = execv + argrest = (args,) + env = environ + + head, tail = path.split(file) + if head: + func(file, *argrest) + return + if 'PATH' in env: + envpath = env['PATH'] + else: + envpath = defpath + PATH = envpath.split(pathsep) + saved_exc = None + saved_tb = None + for dir in PATH: + fullname = path.join(dir, file) + try: + func(fullname, *argrest) + except error, e: + tb = sys.exc_info()[2] + if (e.errno != ENOENT and e.errno != ENOTDIR + and saved_exc is None): + saved_exc = e + saved_tb = tb + if saved_exc: + raise error, saved_exc, saved_tb + raise error, e, tb + +# Change environ to automatically call putenv() if it exists +try: + # This will fail if there's no putenv + putenv +except NameError: + pass +else: + import UserDict + + # Fake unsetenv() for Windows + # not sure about os2 here but + # I'm guessing they are the same. + + if name in ('os2', 'nt'): + def unsetenv(key): + putenv(key, "") + + if name == "riscos": + # On RISC OS, all env access goes through getenv and putenv + from riscosenviron import _Environ + elif name in ('os2', 'nt'): # Where Env Var Names Must Be UPPERCASE + # But we store them as upper case + class _Environ(UserDict.IterableUserDict): + def __init__(self, environ): + UserDict.UserDict.__init__(self) + data = self.data + for k, v in environ.items(): + data[k.upper()] = v + def __setitem__(self, key, item): + putenv(key, item) + self.data[key.upper()] = item + def __getitem__(self, key): + return self.data[key.upper()] + try: + unsetenv + except NameError: + def __delitem__(self, key): + del self.data[key.upper()] + else: + def __delitem__(self, key): + unsetenv(key) + del self.data[key.upper()] + def has_key(self, key): + return key.upper() in self.data + def __contains__(self, key): + return key.upper() in self.data + def get(self, key, failobj=None): + return self.data.get(key.upper(), failobj) + def update(self, dict): + for k, v in dict.items(): + self[k] = v + def copy(self): + return dict(self) + + else: # Where Env Var Names Can Be Mixed Case + class _Environ(UserDict.IterableUserDict): + def __init__(self, environ): + UserDict.UserDict.__init__(self) + self.data = environ + def __setitem__(self, key, item): + putenv(key, item) + self.data[key] = item + def update(self, dict): + for k, v in dict.items(): + self[k] = v + try: + unsetenv + except NameError: + pass + else: + def __delitem__(self, key): + unsetenv(key) + del self.data[key] + def copy(self): + return dict(self) + + + environ = _Environ(environ) + +def getenv(key, default=None): + """Get an environment variable, return None if it doesn't exist. + The optional second argument can specify an alternate default.""" + return environ.get(key, default) +__all__.append("getenv") + +def _exists(name): + try: + eval(name) + return True + except NameError: + return False + +# Supply spawn*() (probably only for Unix) +if _exists("fork") and not _exists("spawnv") and _exists("execv"): + + P_WAIT = 0 + P_NOWAIT = P_NOWAITO = 1 + + # XXX Should we support P_DETACH? I suppose it could fork()**2 + # and close the std I/O streams. Also, P_OVERLAY is the same + # as execv*()? + + def _spawnvef(mode, file, args, env, func): + # Internal helper; func is the exec*() function to use + pid = fork() + if not pid: + # Child + try: + if env is None: + func(file, args) + else: + func(file, args, env) + except: + _exit(127) + else: + # Parent + if mode == P_NOWAIT: + return pid # Caller is responsible for waiting! + while 1: + wpid, sts = waitpid(pid, 0) + if WIFSTOPPED(sts): + continue + elif WIFSIGNALED(sts): + return -WTERMSIG(sts) + elif WIFEXITED(sts): + return WEXITSTATUS(sts) + else: + raise error, "Not stopped, signaled or exited???" + + def spawnv(mode, file, args): + """spawnv(mode, file, args) -> integer + +Execute file with arguments from args in a subprocess. +If mode == P_NOWAIT return the pid of the process. +If mode == P_WAIT return the process's exit code if it exits normally; +otherwise return -SIG, where SIG is the signal that killed it. """ + return _spawnvef(mode, file, args, None, execv) + + def spawnve(mode, file, args, env): + """spawnve(mode, file, args, env) -> integer + +Execute file with arguments from args in a subprocess with the +specified environment. +If mode == P_NOWAIT return the pid of the process. +If mode == P_WAIT return the process's exit code if it exits normally; +otherwise return -SIG, where SIG is the signal that killed it. """ + return _spawnvef(mode, file, args, env, execve) + + # Note: spawnvp[e] is't currently supported on Windows + + def spawnvp(mode, file, args): + """spawnvp(mode, file, args) -> integer + +Execute file (which is looked for along $PATH) with arguments from +args in a subprocess. +If mode == P_NOWAIT return the pid of the process. +If mode == P_WAIT return the process's exit code if it exits normally; +otherwise return -SIG, where SIG is the signal that killed it. """ + return _spawnvef(mode, file, args, None, execvp) + + def spawnvpe(mode, file, args, env): + """spawnvpe(mode, file, args, env) -> integer + +Execute file (which is looked for along $PATH) with arguments from +args in a subprocess with the supplied environment. +If mode == P_NOWAIT return the pid of the process. +If mode == P_WAIT return the process's exit code if it exits normally; +otherwise return -SIG, where SIG is the signal that killed it. """ + return _spawnvef(mode, file, args, env, execvpe) + +if _exists("spawnv"): + # These aren't supplied by the basic Windows code + # but can be easily implemented in Python + + def spawnl(mode, file, *args): + """spawnl(mode, file, *args) -> integer + +Execute file with arguments from args in a subprocess. +If mode == P_NOWAIT return the pid of the process. +If mode == P_WAIT return the process's exit code if it exits normally; +otherwise return -SIG, where SIG is the signal that killed it. """ + return spawnv(mode, file, args) + + def spawnle(mode, file, *args): + """spawnle(mode, file, *args, env) -> integer + +Execute file with arguments from args in a subprocess with the +supplied environment. +If mode == P_NOWAIT return the pid of the process. +If mode == P_WAIT return the process's exit code if it exits normally; +otherwise return -SIG, where SIG is the signal that killed it. """ + env = args[-1] + return spawnve(mode, file, args[:-1], env) + +if _exists("spawnvp"): + # At the moment, Windows doesn't implement spawnvp[e], + # so it won't have spawnlp[e] either. + def spawnlp(mode, file, *args): + """spawnlp(mode, file, *args) -> integer + +Execute file (which is looked for along $PATH) with arguments from +args in a subprocess with the supplied environment. +If mode == P_NOWAIT return the pid of the process. +If mode == P_WAIT return the process's exit code if it exits normally; +otherwise return -SIG, where SIG is the signal that killed it. """ + return spawnvp(mode, file, args) + + def spawnlpe(mode, file, *args): + """spawnlpe(mode, file, *args, env) -> integer + +Execute file (which is looked for along $PATH) with arguments from +args in a subprocess with the supplied environment. +If mode == P_NOWAIT return the pid of the process. +If mode == P_WAIT return the process's exit code if it exits normally; +otherwise return -SIG, where SIG is the signal that killed it. """ + env = args[-1] + return spawnvpe(mode, file, args[:-1], env) + + + __all__.extend(["spawnlp","spawnlpe","spawnv", "spawnve","spawnvp", + "spawnvpe","spawnl","spawnle",]) + + +# Supply popen2 etc. (for Unix) +if _exists("fork"): + if not _exists("popen2"): + def popen2(cmd, mode="t", bufsize=-1): + import popen2 + stdout, stdin = popen2.popen2(cmd, bufsize) + return stdin, stdout + __all__.append("popen2") + + if not _exists("popen3"): + def popen3(cmd, mode="t", bufsize=-1): + import popen2 + stdout, stdin, stderr = popen2.popen3(cmd, bufsize) + return stdin, stdout, stderr + __all__.append("popen3") + + if not _exists("popen4"): + def popen4(cmd, mode="t", bufsize=-1): + import popen2 + stdout, stdin = popen2.popen4(cmd, bufsize) + return stdin, stdout + __all__.append("popen4") + +import copy_reg as _copy_reg + +def _make_stat_result(tup, dict): + return stat_result(tup, dict) + +def _pickle_stat_result(sr): + (type, args) = sr.__reduce__() + return (_make_stat_result, args) + +try: + _copy_reg.pickle(stat_result, _pickle_stat_result, _make_stat_result) +except NameError: # stat_result may not exist + pass + +def _make_statvfs_result(tup, dict): + return statvfs_result(tup, dict) + +def _pickle_statvfs_result(sr): + (type, args) = sr.__reduce__() + return (_make_statvfs_result, args) + +try: + _copy_reg.pickle(statvfs_result, _pickle_statvfs_result, + _make_statvfs_result) +except NameError: # statvfs_result may not exist + pass Modified: pypy/trunk/src/pypy/appspace/types.py ============================================================================== --- pypy/trunk/src/pypy/appspace/types.py (original) +++ pypy/trunk/src/pypy/appspace/types.py Sun Nov 30 18:53:12 2003 @@ -1,5 +1,8 @@ """Appspace types module. +!! This file has been copied practicaly verbatim from the CPython source. +!! See http://www.python.org/2.3.2/license.html for licensing info. + Define names for all type symbols known in the standard interpreter. Types that are part of optional modules (e.g. array) are not listed. @@ -24,6 +27,10 @@ pass FloatType = float try: + BooleanType = bool +except NameError: + pass +try: ComplexType = complex except NameError: pass @@ -79,6 +86,9 @@ ModuleType = type(sys) try: FileType = file +except NameError: + pass +try: XRangeType = type(xrange(0)) except NameError: pass @@ -100,5 +110,9 @@ EllipsisType = type(Ellipsis) #DictProxyType = type(TypeType.__dict__) +try: + NotImplementedType = type(NotImplemented) +except NameError: + pass del sys, _f, _C, _x#, generators # Not for export Modified: pypy/trunk/src/pypy/module/__init__.py ============================================================================== --- pypy/trunk/src/pypy/module/__init__.py (original) +++ pypy/trunk/src/pypy/module/__init__.py Sun Nov 30 18:53:12 2003 @@ -28,4 +28,9 @@ ('os_modules','Mac',_std_spaces), ('os_modules','Ce',_std_spaces), ('os_modules','Riscos',_std_spaces), + ('mathmodule','Math',_std_spaces), + ('_randommodule','RandomHelper',_std_spaces), + ('cStringIOmodule','CStringIO',_std_spaces), + ('itertoolsmodule','Itertools',_std_spaces), + ('_sremodule','SreHelper',_std_spaces), ] Added: pypy/trunk/src/pypy/module/_randommodule.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/module/_randommodule.py Sun Nov 30 18:53:12 2003 @@ -0,0 +1,20 @@ +"""Bootstrap the builtin _random module. + +""" +import sys + +from pypy.interpreter.extmodule import ExtModule + +_names = sys.builtin_module_names + +# We use the second (metaclassish) meaning of type to construct a subclass +# of ExtModule - can't modify some attributes (like __doc__) after class +# creation, and wrapping code does not properly look at instance variables. + +def RandomHelper(space): + if '_random' in _names: + import _random + _randomhelper = type('_random', (ExtModule,), _random.__dict__) + return _randomhelper(space) + else: + return None Added: pypy/trunk/src/pypy/module/_sremodule.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/module/_sremodule.py Sun Nov 30 18:53:12 2003 @@ -0,0 +1,17 @@ +"""Bootstrap the builtin _sre module. + +""" +import sys + +from pypy.interpreter.extmodule import ExtModule + +_names = sys.builtin_module_names + +# We use the second (metaclassish) meaning of type to construct a subclass +# of ExtModule - can't modify some attributes (like __doc__) after class +# creation, and wrapping code does not properly look at instance variables. + +def SreHelper(space): + import _sre + _srehelper = type('_sre', (ExtModule,), _sre.__dict__) + return _srehelper(space) Modified: pypy/trunk/src/pypy/module/builtin.py ============================================================================== --- pypy/trunk/src/pypy/module/builtin.py (original) +++ pypy/trunk/src/pypy/module/builtin.py Sun Nov 30 18:53:12 2003 @@ -94,9 +94,13 @@ elif loc is None: loc = glob f = file(filename) - source = f.read() - f.close() - exec source in glob, loc + try: + source = f.read() + finally: + f.close() + #Don't exec the source directly, as this loses the filename info + co = compile(source, filename, 'exec') + exec co in glob, loc ####essentially implemented by the objectspace def abs(self, w_val): @@ -383,6 +387,9 @@ except AttributeError: return False + def app_callable(self, ob): + return hasattr(ob, '__call__') + def app_dir(self, *args): """dir([object]) -> list of strings Added: pypy/trunk/src/pypy/module/cStringIOmodule.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/module/cStringIOmodule.py Sun Nov 30 18:53:12 2003 @@ -0,0 +1,20 @@ +"""Bootstrap the builtin cStringIO module. + +""" +import sys + +from pypy.interpreter.extmodule import ExtModule + +_names = sys.builtin_module_names + +# We use the second (metaclassish) meaning of type to construct a subclass +# of ExtModule - can't modify some attributes (like __doc__) after class +# creation, and wrapping code does not properly look at instance variables. + +def CStringIO(space): + if 'cStringIO' in _names: + import cStringIO + _cStringIO = type('cStringIO', (ExtModule,), cStringIO.__dict__) + return _cStringIO(space) + else: + return None Added: pypy/trunk/src/pypy/module/itertoolsmodule.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/module/itertoolsmodule.py Sun Nov 30 18:53:12 2003 @@ -0,0 +1,20 @@ +"""Bootstrap the builtin itertools module. + +""" +import sys + +from pypy.interpreter.extmodule import ExtModule + +_names = sys.builtin_module_names + +# We use the second (metaclassish) meaning of type to construct a subclass +# of ExtModule - can't modify some attributes (like __doc__) after class +# creation, and wrapping code does not properly look at instance variables. + +def Itertools(space): + if 'itertools' in _names: + import itertools + _itertools = type('itertools', (ExtModule,), itertools.__dict__) + return _itertools(space) + else: + return None Added: pypy/trunk/src/pypy/module/mathmodule.py ============================================================================== --- (empty file) +++ pypy/trunk/src/pypy/module/mathmodule.py Sun Nov 30 18:53:12 2003 @@ -0,0 +1,20 @@ +"""Bootstrap the builtin math module. + +""" +import sys + +from pypy.interpreter.extmodule import ExtModule + +_names = sys.builtin_module_names + +# We use the second (metaclassish) meaning of type to construct a subclass +# of ExtModule - can't modify some attributes (like __doc__) after class +# creation, and wrapping code does not properly look at instance variables. + +def Math(space): + if 'math' in _names: + import math + _math = type('math', (ExtModule,), math.__dict__) + return _math(space) + else: + return None Modified: pypy/trunk/src/pypy/module/sysmodule.py ============================================================================== --- pypy/trunk/src/pypy/module/sysmodule.py (original) +++ pypy/trunk/src/pypy/module/sysmodule.py Sun Nov 30 18:53:12 2003 @@ -20,6 +20,10 @@ self.w_path = space.newlist([appdir] + [p for p in cpy_sys.path if p!= pypydir]) self.w_modules = space.newdict([]) self.w_builtin_module_names = space.newlist([]) + self.w_warnoptions = space.newlist([space.wrap(i) for i in cpy_sys.warnoptions]) + # XXX - Replace with appropriate PyPy version numbering + self.w_hexversion = space.wrap(cpy_sys.hexversion) + self.w_platform = space.wrap(cpy_sys.platform) ExtModule.__init__(self, space) stdout = cpy_sys.stdout Modified: pypy/trunk/src/pypy/module/test/test_builtin.py ============================================================================== --- pypy/trunk/src/pypy/module/test/test_builtin.py (original) +++ pypy/trunk/src/pypy/module/test/test_builtin.py Sun Nov 30 18:53:12 2003 @@ -95,6 +95,19 @@ self.assert_(hasattr(self.space.builtin, 'xrange')) self.assertEquals(self.space.builtin.xrange(3).stop, 3) + def test_callable(self): + class Call: + def __call__(self, a): + return a+2 + self.failIf(not callable(Call()), + "Builtin function 'callable' misreads callable object") + def test_uncallable(self): + class NoCall: + pass + self.failIf(callable(NoCall()), + "Builtin function 'callable' misreads uncallable object") + + if __name__ == '__main__': test.main() Modified: pypy/trunk/src/pypy/module/test/test_sysmodule.py ============================================================================== --- pypy/trunk/src/pypy/module/test/test_sysmodule.py (original) +++ pypy/trunk/src/pypy/module/test/test_sysmodule.py Sun Nov 30 18:53:12 2003 @@ -29,6 +29,18 @@ import sys self.failUnless(hasattr(sys, 'builtin_module_names'), "sys.builtin_module_names gone missing") + def test_warnoptions_exists(self): + import sys + self.failUnless(hasattr(sys, 'warnoptions'), + "sys.warnoptions gone missing") + def test_hexversion_exists(self): + import sys + self.failUnless(hasattr(sys, 'hexversion'), + "sys.hexversion gone missing") + def test_platform_exists(self): + import sys + self.failUnless(hasattr(sys, 'platform'), "sys.platform gone missing") + def test_sys_in_modules(self): import sys modules = sys.modules