[pypy-svn] r14764 - pypy/dist/pypy/documentation
hpk at codespeak.net
hpk at codespeak.net
Tue Jul 19 14:06:17 CEST 2005
Author: hpk
Date: Tue Jul 19 14:06:15 2005
New Revision: 14764
Modified:
pypy/dist/pypy/documentation/_ref.txt
pypy/dist/pypy/documentation/coding-guide.txt
pypy/dist/pypy/documentation/index.txt
pypy/dist/pypy/documentation/interpreter.txt
Log:
issue66 in-progress
implemented a first draft of the interpreter documentation
still missing bits and pieces (it's a bit hard to draw
the lines).
also fixed some links here and there and inserted the
bytecode interpreter into the index.txt.
feedback already welcome.
Modified: pypy/dist/pypy/documentation/_ref.txt
==============================================================================
--- pypy/dist/pypy/documentation/_ref.txt (original)
+++ pypy/dist/pypy/documentation/_ref.txt Tue Jul 19 14:06:15 2005
@@ -8,6 +8,14 @@
.. _`documentation/website/`: http://codespeak.net/svn/pypy/dist/pypy/documentation/website
.. _`pypy/interpreter`:
.. _`interpreter/`: http://codespeak.net/svn/pypy/dist/pypy/interpreter
+.. _`pypy/interpreter/argument.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/argument.py
+.. _`pypy/interpreter/function.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/function.py
+.. _`pypy/interpreter/gateway.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/gateway.py
+.. _`pypy/interpreter/generator.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/generator.py
+.. _`pypy/interpreter/mixedmodule.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/mixedmodule.py
+.. _`pypy/interpreter/nestedscope.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/nestedscope.py
+.. _`pypy/interpreter/pyopcode.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/pyopcode.py
+.. _`pypy/interpreter/typedef.py`: http://codespeak.net/svn/pypy/dist/pypy/interpreter/typedef.py
.. _`lib/`:
.. _`pypy/lib/`: http://codespeak.net/svn/pypy/dist/pypy/lib
.. _`lib/test2/`:
@@ -26,8 +34,8 @@
.. _`pypy/objspace/std`:
.. _`objspace/std/`: http://codespeak.net/svn/pypy/dist/pypy/objspace/std
.. _`objspace/thunk.py`: http://codespeak.net/svn/pypy/dist/pypy/objspace/thunk.py
-.. _`pypy/objspace/trace.py`:
-.. _`objspace/trace.py`: http://codespeak.net/svn/pypy/dist/pypy/objspace/trace.py
+.. _`objspace/trace.py`:
+.. _`pypy/objspace/trace.py`: http://codespeak.net/svn/pypy/dist/pypy/objspace/trace.py
.. _`rpython/`: http://codespeak.net/svn/pypy/dist/pypy/rpython
.. _`rpython/lltype.py`: http://codespeak.net/svn/pypy/dist/pypy/rpython/lltype.py
.. _`rpython/rint.py`: http://codespeak.net/svn/pypy/dist/pypy/rpython/rint.py
@@ -40,7 +48,7 @@
.. _`pypy/translator`:
.. _`translator/`: http://codespeak.net/svn/pypy/dist/pypy/translator
.. _`pypy/translator/annrpython.py`: http://codespeak.net/svn/pypy/dist/pypy/translator/annrpython.py
-.. _`translator/genc/`: http://codespeak.net/svn/pypy/dist/pypy/translator/genc
+.. _`translator/c/`: http://codespeak.net/svn/pypy/dist/pypy/translator/c
.. _`translator/java/`: http://codespeak.net/svn/pypy/dist/pypy/translator/java
.. _`translator/llvm/`: http://codespeak.net/svn/pypy/dist/pypy/translator/llvm
.. _`translator/tool/`: http://codespeak.net/svn/pypy/dist/pypy/translator/tool
\ No newline at end of file
Modified: pypy/dist/pypy/documentation/coding-guide.txt
==============================================================================
--- pypy/dist/pypy/documentation/coding-guide.txt (original)
+++ pypy/dist/pypy/documentation/coding-guide.txt Tue Jul 19 14:06:15 2005
@@ -483,6 +483,7 @@
``space.call_method(w_dict, 'iterkeys')`` returns a wrapped
iterable that you can decode with ``space.unpackiterable()``.
+.. _`applevel-exceptions`:
Application-level exceptions
----------------------------
Modified: pypy/dist/pypy/documentation/index.txt
==============================================================================
--- pypy/dist/pypy/documentation/index.txt (original)
+++ pypy/dist/pypy/documentation/index.txt Tue Jul 19 14:06:15 2005
@@ -14,6 +14,9 @@
`object spaces`_ discusses the object space interface
and several implementations.
+`bytecode interpreter`_ explains the basic mechanisms
+of the bytecode interpreter and virtual machine.
+
`translation`_ offers the beginnings of documentation
about our low level code generator backends.
@@ -35,7 +38,7 @@
.. _`revision report`: http://codespeak.net/pypy/rev/current
.. _`getting started`: getting_started.html
.. _`theory`: theory.html
-
+.. _`bytecode interpreter`: interpreter.html
.. _`directory reference`:
@@ -56,7 +59,7 @@
`documentation/website/`_ text versions of the navigation webpages
-`interpreter/`_ bytecode interpreter and related objects (frames, functions, modules,...)
+`interpreter/`_ `bytecode interpreter`_ and related objects (frames, functions, modules,...)
`lib/`_ PyPy's wholesale reimplementations of CPython modules_
@@ -97,7 +100,7 @@
`translator/`_ translation_ backends and support code
-`translator/genc/`_ the `GenC backend`_ producing a CPython C-extension
+`translator/c/`_ the `GenC backend`_ producing a CPython C-extension
module from a given RPython program.
`translator/java/`_ experimental code to utilize Java for annotation
@@ -114,6 +117,7 @@
level to interpreterlevel`_ code.
============================ ===========================================
+.. _`bytecode interpreter`: interpreter.html
.. _`translating application level to interpreterlevel`: translation.html#python-back-end
.. _documentation: index.html
.. _`Testing in PyPy`: coding-guide.html#testing-in-pypy
Modified: pypy/dist/pypy/documentation/interpreter.txt
==============================================================================
--- pypy/dist/pypy/documentation/interpreter.txt (original)
+++ pypy/dist/pypy/documentation/interpreter.txt Tue Jul 19 14:06:15 2005
@@ -12,40 +12,47 @@
============
This document describes the implementation of PyPy's
-`Bytecode Interpreter`_ and related basic VM functionalities.
+`Bytecode Interpreter`_ and related Virtual Machine functionalities.
Overview
============
PyPy's bytecode interpreter has a structure reminiscent of
-CPython's VM implementation: Source code is parsed and
-compiled into code objects which encapsulate information about
-their respective functions, class and module body source
-codes. Interpreting such code objects means instantiating and
+CPython's Virtual Machine: Source code is parsed and compiled
+into code objects which encapsulate information about their
+respective functions, class and module body source codes.
+Interpreting such code objects means instantiating and
initializing a `Frame class`_ and then calling its
``frame.eval()`` method. This entry point then interprets
each bytecode.
-The bytecode interpreter is only responsible for implementing
-control flow and putting and pulling black box objects to and
-from the value stack. It does not know how to perform
-operations on those black box (`wrapped`_) objects for which
-it defers to the `object space`_. In order to implement a
-branch in a program's execution, however, it needs to gain
-minimal knowledge about a wrapped object. Thus, each object
-space offers a ``is_true(w_obj)`` method which returns an
-interpreter-level boolean value.
+CPython as well as PyPy are stack-based virtual machines, i.e.
+they don't have registers but put object to and pull objects
+from a stack. The bytecode interpreter is only responsible
+for implementing control flow and putting and pulling black
+box objects to and from the value stack. It does not know how
+to perform operations on those black box (`wrapped`_) objects
+for which it delegates to the `object space`_. In order to
+implement a branch in a program's execution, however, it needs
+to gain minimal knowledge about a wrapped object. Thus, each
+object space has to offer a ``is_true(w_obj)`` operation which
+returns an interpreter-level boolean value.
For the understanding of the interpreter's inner workings it
is crucial to recognize the concepts of `interpreter-level and
application-level`_ code. In short, interpreter-level is executed
directly on the machine and invoking application-level functions
-leads to the full prescribed bytecode interpretation indirection.
+leads to an bytecode interpretation indirection. However,
+special care must be taken regarding exceptions because
+application level exceptions are wrapped into ``OperationErrors``
+which are thus distinguished from plain interpreter-level exceptions.
+See `application level exceptions`_ for some more information
+on ``OperationErrors``.
The interpreter implementation offers mechanisms to allow a
caller to be unaware if a particular function invocation leads
to bytecode interpretation or is executed directly at
-intepreter-level. The two basic kinds of `Gateway classes`_
+interpreter-level. The two basic kinds of `Gateway classes`_
expose either an interpreter-level function to
application-level execution (``interp2app``) or allow
transparent invocation of application-level helpers
@@ -54,51 +61,319 @@
Another task of the interpreter is to expose its basic
code, frame, module and function objects to application-level
code. Such runtime introspection and modification abilities are
-implemented via descriptors.
+implemented via `interpreter descriptors`_ (also see Raymond Hettingers
+`how-to guide for descriptors`_ in Python, PyPy uses this model extensively).
-A significant complexity lies in the `argument parsing`_
-implementation. Python as a language offers very flexible
-ways of providing and receiving arguments for a particular
-function invocation. Not only does it take care to get this
-right, it also presents difficulties for the `annotation
+A significant complexity lies in `argument parsing`_. Python as a
+language offers very flexible ways of providing and receiving arguments
+for a particular function invocation. Not only does it take special care
+to get this right, it also presents difficulties for the `annotation
pass`_ which performs a whole-program analysis on the
-bytecode interpreter, argument parsing and gateway-code
-in order to infer the types of all values.
+bytecode interpreter, argument parsing and gatewaying code
+in order to infer the types of all flowing values.
-It is for this reason that PyPy sometimes resorts to generate
+It is for this reason that PyPy resorts to generate
specialized frame classes and functions at `initialization
time`_ in order to let the annotator only see static program
flows with homogenous name-value assignments on e.g. function
invocations.
+.. _`how-to guide for descriptors`: http://users.rcn.com/python/download/Descriptor.htm
.. _`annotation pass`: translation.html#the-annotation-pass
.. _`initialization time`: architecture.html#initialization-time
.. _`interpreter-level and application-level`: architecture.html#interpreter-level
.. _`wrapped`: coding-guide.html#wrapping-rules
.. _`object space`: architecture.html#objectspace
+.. _`application level exceptions`: coding-guide.html#applevel-exceptions
+
+
+Interpreter Implementation Classes
+======================================
.. _`Frame class`:
+.. _`Frame`:
Frame classes
-========================
-XXX
+-----------------
-the eval loop
----------------------
+The concept of Frames is pervasive in executing programs and
+virtual machines in particular. They are sometimes called
+*execution frame* because they hold crucial information
+regarding the execution of a Code_ object, which in turn is
+often directly related to a Python `Function`_. Frame
+instances hold the following state:
+
+- the local scope holding name-value bindings, usually implemented
+ via a "fast scope" which is an array of wrapped objects
+
+- a blockstack which has (nested) information regarding the
+ control flow of a function (such as ``while`` and ``try`` constructs)
+
+- a value stack where bytecode interpretation pulls object
+ from and puts results on.
+
+- a reference to the *globals* dictionary, containing
+ module-level name-value bindings
+
+- debugging information from which a current line-number and
+ file location can be constructed for tracebacks
+
+Moreover the Frame class itself has a number of methods which implement
+the actual bytecodes found in a code object. In fact, PyPy already constructs
+four specialized Frame class variants depending on the code object:
+
+- PyInterpFrame (in `pypy/interpreter/pyopcode.py`_) for
+ basic simple code objects (not involving generators or nested scopes)
+
+- PyNestedScopeFrame (in `pypy/interpreter/nestedscope.py`_)
+ for code objects that reference nested scopes, inherits from PyInterpFrame
+
+- PyGeneratorFrame (in `pypy/interpreter/generator.py`_)
+ for code objects that yield values to the caller, inherits from PyInterpFrame
+
+- PyNestedScopeGeneratorFrame for code objects that reference
+ nested scopes and yield values to the caller, inherits from both PyNestedScopeFrame
+ and PyGeneratorFrame
+
+.. _Code:
+
+Code Class
+------------
+
+PyPy's code objects contain the same information found in CPython's code objects.
+the differ from Function_ objects in that they are only immutable representations
+of source code and don't contain execution state or references to the execution
+environment found in `Frames`. Frames and Functions have references
+to a code object. Here is a list of Code attributes:
+
+* ``co_flags`` flags if this code object has nested scopes/generators
+* ``co_stacksize`` flags if this code object has nested scopes/generators
+* ``co_code`` the actual bytecode string
+
+* ``co_argcount`` number of arguments this code object expects
+* ``co_varnames`` a tuple of all argument names pass to this code object
+* ``co_nlocals`` number of local variables
+* ``co_names`` a tuple of all names used in the code object
+* ``co_consts`` a tuple of prebuilt constant objects ("literals") used in the code object
+* ``co_cellvars`` a tuple of Cells containing values for access from nested scopes
+* ``co_freevars`` a tuple of Cell names from "above" scopes
+
+* ``co_filename`` source file this code object was compiled from
+* ``co_firstlineno`` the first linenumber of the code object in its source file
+* ``co_name`` name of the code object (often the function name)
+* ``co_lnotab`` a helper table to compute the line-numbers corresponding to bytecodes
+
+In PyPy, code objects also have the responsibility of creating their Frame_ objects
+via the `'create_frame()`` method. With proper parser and compiler support this should
+allow to create custom Frame objects extending the execution of functions
+in various ways. The several Frame_ classes already utilize this flexibility
+in order to implement Generators and Nested Scopes.
+
+.. _Function:
+
+Function and Method classes
+----------------------------
+
+The PyPy ``Function`` class (in `pypy/interpreter/function.py`_)
+represents a Python function. A ``Function`` carries the following
+main attributes:
+
+* ``func_doc`` the docstring (or None)
+* ``func_name`` the name of the function
+* ``func_code`` the Code_ object representing the function source code
+* ``func_defaults`` default values for the function (built at function definition time)
+* ``func_dict`` dictionary for additional (user-defined) function attributes
+* ``func_globals`` reference to the globals dictionary
+* ``func_closure`` a tuple of Cell references
+
+``Functions`` classes also provide a ``__get__`` descriptor which creates a Method
+object holding a binding to an instance or a class. Finally, ``Functions``
+and ``Methods`` both offer a ``call_args()`` method which executes
+the function given an `Arguments`_ class instance.
+.. _Arguments:
.. _`argument parsing`:
-Argument Parsing
-=======================
+Arguments Class
+--------------------
+
+The Argument class (in `pypy/interpreter/argument.py`_) is
+responsible for parsing arguments passed to functions.
+Python has rather complex argument-passing concepts:
+
+- positional arguments
+
+- keyword arguments specified by name
+
+- default values for positional arguments, defined at function
+ definition time
+
+- "star args" allowing a function to accept remaining
+ positional arguments
+
+- "star keyword args" allow a function to accept additional
+ arbitrary name-value bindings
-XXX
+Moreover, a Function_ object can get bound to a class or instance
+in which case the first argument to the underlying function becomes
+the bound object. The ``Arguments`` provides means to allow all
+this argument parsing and also cares for error reporting.
+
+
+.. _`Module`:
+
+Module Class
+-------------------
+
+A ``Module`` instance represents execution state usually constructed
+from executing the module's source file. In addition to such a module's
+global ``__dict__`` dictionary it has the following application level
+attributes:
+
+* ``__doc__`` the docstring of the module
+* ``__file__`` the source filename from which this module was instantiated
+* ``__path__`` state used for relative imports
+
+Apart from the basic Module used for importing application-level files
+there is a more refined ``MixedModule`` class (see `pypy/interpreter/mixedmodule.py`_)
+which allows to define name-value bindings both at application level
+and an intepreter level. See the ``__builtin__`` module's
+`pypy/module/__builtin__/__init__.py`_ file for an example
+and the higher level `chapter on Modules in the coding guide`_.
+
+.. _`__builtin__ module`: http://codespeak.net/svn/pypy/dist/pypy/module/
+.. _`chapter on Modules in the coding guide`: coding-guide.html#modules
.. _`Gateway classes`:
Gateway classes
-=======================
-
-XXX
+----------------------
+A unique PyPy property is the ability to easily cross the barrier
+between interpreted code and machine-level code. Both codes are
+implemented via Python source code but they usually look differently.
+Be aware that the according code (in `pypy/interpreter/gateway.py`_)
+for crossing the barrier in both directions is somewhat involved, mostly
+due to the fact that the type-infering annotator needs to keep
+track of the types of objects flowing across those barriers.
+
+Making interpreter-level functions available at application-level
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+In order to make an interpreter-level function available at
+application level, one invokes ``pypy.interpreter.gateway.interp2app(func)``.
+Such a function usually takes a ``space`` argument and any number
+of positional arguments. Additionally, such functions can define
+an ``unwrap_spec`` telling the ``interp2app`` logic how
+application-level provided arguments should be unwrapped
+before the actual interpreter-level function is invoked.
+For example, `interpreter descriptors`_ such as the ``Module.__new__``
+method for allocating and constructing a Module instance is
+defined with this code::
+
+ Module.typedef = TypeDef("module",
+ __new__ = interp2app(Module.descr_module__new__.im_func,
+ unwrap_spec=[ObjSpace, W_Root, Arguments]),
+ __init__ = interp2app(Module.descr_module__init__),
+ # module dictionaries are readonly attributes
+ __dict__ = GetSetProperty(descr_get_dict, cls=Module),
+ __doc__ = 'module(name[, doc])\n\nCreate a module object...'
+ )
+
+The actual ``Module.descr_module__new__`` interpreter-level method
+referenced from the ``__new__`` keyword argument above is defined
+like this::
+
+ def descr_module__new__(space, w_subtype, __args__):
+ module = space.allocate_instance(Module, w_subtype)
+ Module.__init__(module, space, None)
+ return space.wrap(module)
+
+Summarizing, the ``interp2app`` mechanism takes care to route
+an application level access or call to an internal interpreter-level
+object appropriately to the descriptor, providing enough precision
+and hints to keep the type-infering annotator happy.
+
+
+Calling into application level code from interpreter-level
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Application level code is `often preferable`_. Therefore,
+we often like to invoke application level code from interpreter-level.
+This is done via the Gateway's ``app2interp`` mechanism
+which we usually invoke at definition time in a module.
+It generates a hook which looks like an intepreter-level
+function accepting a space and an arbitrary number of arguments.
+When calling a function at interpreter-level the caller side
+does usually not need to be aware if its invoked function
+is run through the PyPy interpreter or if it will directly
+execute on the machine (after translation).
+
+Here is an example showing how we implement the Metaclass
+finding algorithm of the Python language in PyPy::
+
+ app = gateway.applevel(r'''
+ def find_metaclass(bases, namespace, globals, builtin):
+ if '__metaclass__' in namespace:
+ return namespace['__metaclass__']
+ elif len(bases) > 0:
+ base = bases[0]
+ if hasattr(base, '__class__'):
+ return base.__class__
+ else:
+ return type(base)
+ elif '__metaclass__' in globals:
+ return globals['__metaclass__']
+ else:
+ try:
+ return builtin.__metaclass__
+ except AttributeError:
+ return type
+ ''', filename=__file__)
+
+ find_metaclass = app.interphook('find_metaclass')
+
+The ``find_metaclass`` interpreter-level hook is invoked
+with five arguments from the ``BUILD_CLASS`` opcode implementation
+in `pypy/interpreter/pyopcode.py`_::
+
+ def BUILD_CLASS(f):
+ w_methodsdict = f.valuestack.pop()
+ w_bases = f.valuestack.pop()
+ w_name = f.valuestack.pop()
+ w_metaclass = find_metaclass(f.space, w_bases,
+ w_methodsdict, f.w_globals,
+ f.space.wrap(f.builtin))
+ w_newclass = f.space.call_function(w_metaclass, w_name,
+ w_bases, w_methodsdict)
+ f.valuestack.push(w_newclass)
+
+Note that at a later point we might rewrite the ``find_metaclass``
+implementation at interpreter-level but we would not have
+to modify the calling sides at all.
+
+.. _`often preferable`: architecture.html#app-preferable
+.. _`interpreter descriptors`:
+
+Introspection and Descriptors
+------------------------------
+
+Python traditionally has a very far-reaching introspection model
+for interpreter related objects. In PyPy and in CPython read
+and write accesses to such objects are routed to descriptors.
+Of course, in CPython those are implemented in ``C`` while in
+PyPy they are implemented in interpreter-level Python code.
+
+All instances of a Function_, Code_, Frame_ or Module_ class
+are ``Wrappable`` instances and can thus be represented at
+application level. These days, a PyPy object space needs to
+work with a basic descriptor lookup when it encounters
+accesses to an interpreter-level object. An object space ask
+a wrapped object for its type via a ``getclass`` method and then
+calls its ``lookup(name)`` function in order to receive a descriptor function.
+Most of PyPy's internal object descriptors are defined at the
+end of `pypy/interpreter/typedef.py`_. You can use this as a reference
+for the exact attributes visible at application level.
+.. include:: _ref.txt
More information about the Pypy-commit
mailing list