[pypy-svn] r36997 - pypy/dist/pypy/doc
arigo at codespeak.net
arigo at codespeak.net
Fri Jan 19 15:51:02 CET 2007
Author: arigo
Date: Fri Jan 19 15:50:59 2007
New Revision: 36997
Modified:
pypy/dist/pypy/doc/_ref.txt
pypy/dist/pypy/doc/rctypes.txt
Log:
Updated the ctypes documentation to mention the new approach.
Modified: pypy/dist/pypy/doc/_ref.txt
==============================================================================
--- pypy/dist/pypy/doc/_ref.txt (original)
+++ pypy/dist/pypy/doc/_ref.txt Fri Jan 19 15:50:59 2007
@@ -66,10 +66,13 @@
.. _`rlib/`: ../../pypy/rlib
.. _`pypy/rlib/objectmodel.py`: ../../pypy/rlib/objectmodel.py
.. _`pypy/rlib/rarithmetic.py`: ../../pypy/rlib/rarithmetic.py
+.. _`pypy/rlib/rctypes/rctypesobject.py`: ../../pypy/rlib/rctypes/rctypesobject.py
+.. _`pypy/rlib/rctypes/test/test_rctypesobject.py`: ../../pypy/rlib/rctypes/test/test_rctypesobject.py
.. _`pypy/rlib/test`: ../../pypy/rlib/test
.. _`pypy/rpython`:
.. _`pypy/rpython/`:
.. _`rpython/`: ../../pypy/rpython
+.. _`pypy/rpython/controllerentry.py`: ../../pypy/rpython/controllerentry.py
.. _`pypy/rpython/extfunctable.py`: ../../pypy/rpython/extfunctable.py
.. _`rpython/lltypesystem/`: ../../pypy/rpython/lltypesystem
.. _`pypy/rpython/lltypesystem/lltype.py`:
Modified: pypy/dist/pypy/doc/rctypes.txt
==============================================================================
--- pypy/dist/pypy/doc/rctypes.txt (original)
+++ pypy/dist/pypy/doc/rctypes.txt Fri Jan 19 15:50:59 2007
@@ -2,18 +2,19 @@
RCtypes
============================================================
-.. contents::
+.. contents:: :depth: 3
.. sectnum::
-Using RCtypes
-================
+------------------------------------------------------------
+ Using RCtypes
+------------------------------------------------------------
ctypes_ is a library for CPython that has been around for some years,
and that is now integrated in Python 2.5. It allows Python programs to
directly perform calls to external libraries written in C, by declaring
function prototypes and all necessary data structures dynamically in
-Python.
+Python. It is based on the platform-dependent libffi_ library.
*RCtypes* is ctypes restricted to be used from RPython programs.
Basically, an RPython program that is going to be translated to C or a
@@ -28,13 +29,15 @@
library -- not only from the PyPy interpreter, but from any program or
extension module that is written in RPython.
-This document assumes some basic familiarity with ctypes.
-Look up the `ctypes documentation`_, or see a few notes below about
-the `CPython's ctype's behaviour`_.
+This document assumes some basic familiarity with ctypes. It is
+recommended to first look up the `ctypes documentation`_. As an
+appendix to this document we also point out the `Finest issues in the
+behavior of ctypes`_, which are not explicitly documented anywhere else
+to the best of our knowledge.
Restrictions
-------------
+============
The main restriction that RCtypes adds over ctypes is similar to the
general restriction that RPython_ adds over Python: you cannot be "too
@@ -53,25 +56,27 @@
added at a later time), pointers, structures, arrays, external functions.
The following ctypes functions are supported at run-time:
create_string_buffer(), pointer(), POINTER(), sizeof(), cast(). (Adding
-support for more is easy.) Free union are *not* supported. There is
-some support for callbacks and a few more obscure ctypes features, but
+support for more is easy.) There is some support for Unions,
+and for callbacks and a few more obscure ctypes features, but
e.g. custom allocators and return-value-error-checkers are not
supported.
-Remember to use ``None`` as the return type in the function declaration if it returns void.
+Remember to use ``None`` as the return type in the function declaration
+if it returns void.
There is special support for the special variable ``errno`` of C: from
-the RPython program, use ``pypy.rpython.rctypes.aerrno.geterrno()`` to
-read it.
+the RPython program, access it with the helper function
+``pypy.rpython.rctypes.aerrno.geterrno()``.
Note that the support for the POINTER() function is an exception to the
rule that types should not be manipulated at run-time. You can only
call it with a constant type, as in ``POINTER(c_int)``.
-*New:* an expression like ``c_int * n`` can appear at run-time. It can
+Another exception is that
+an expression like ``c_int * n`` can appear at run-time. It can
be used to create variable-sized array instances, i.e. arrays whose
length is not a static constant, as in ``my_array = (c_int * n)()``.
-Similarly, the create_string_buffer() function returns a variable-size
+Similarly, the create_string_buffer() function returns a variable-sized
array of chars.
NOTE: in order to translate an RPython program using ctypes, the module
@@ -84,12 +89,12 @@
.. _installation:
ctypes version and platform notes
----------------------------------
+=================================
-As of this time, only `ctypes-0.9.9.6`_ is known to work correctly
-with RCtypes.
+`ctypes-0.9.9.6`_ or later is required.
-On Mac OSX 10.3 you need to change the ``RTLD_LOCAL`` default
+On Mac OSX 10.3 (at least with ctypes-0.9.9.6)
+you need to change the ``RTLD_LOCAL`` default
in ctypes/__init__.py line 293 to::
def __init__(self, name, mode=RTLD_GLOBAL, handle=None):
@@ -98,9 +103,33 @@
when trying to load the C library.
+------------------------------------------------------------
+ Implementation design
+------------------------------------------------------------
+
+We have experimented with two different implementation approaches. The
+first one is relatively mature, but does not do the right thing about
+memory management issues in the most advanced cases. Moreover, it
+prevents the RPython program to be compiled with a Moving Garbage
+Collector, which has been the major factor stopping us from
+experimenting with advanced GCs so far. It is described in the chapter
+`RCtypes implemented in the RTyper`_.
+
+The alternative implementation of RCtypes is work in progress; its
+approach is described in the chapter `RCtypes implemented via a pure
+RPython interface`_.
+
+
+RCtypes implemented in the RTyper
+=================================
+
+The currently available implementation of RCtypes works by integrating
+itself within the annotation and most importantly RTyping_ process of
+the translation toolchain: the ctypes objects that the RPython program
+uses, and the operations it performs on them, are individually replaced
+during RTyping by sequences of low-level operations performing the
+equivalent operation in a C-like language.
-Implementation design
-=====================
Annotation
----------
@@ -300,11 +329,127 @@
keepalives.
+RCtypes implemented via a pure RPython interface
+================================================
+
+As of January 2007, a second implementation of RCtypes is being
+developed. The basic conclusion on the work on the first implementation
+is that it is quite tedious to have to write code that generates
+detailed low-level operations for each high-level ctypes operation - and
+most importantly, it is not flexible enough for our purposes. The major
+issue is that problems were recently found in the corner cases of memory
+management, i.e. when ctypes objects should keep other ctypes object
+alive and for how long. Fixing this would involve some pervasive
+changes in the low-level representation of ctypes objects, which would
+require all the RCtypes RTyping code to be updated. Similarly, to be
+able to use a Moving Garbage Collector, the memory that is visible to
+the external C code need to be separated from the GC-managed memory that
+is used for the RCtypes objects themselves; this would also require a
+complete upgrade of all the RTyping code.
+
+
+The rctypesobject RPython library
+---------------------------------
+
+To solve this, we started by writing a complete RPython library that
+offers the same functionality as ctypes, but with a more regular,
+RPython-compatible interface. This library is in the
+`pypy/rlib/rctypes/rctypesobject.py`_ module. All operations use either
+static or instance methods on classes following a structure similar to
+the ctypes classes. The flexibility of writing a normal RPython library
+allowed us to use internal support classes freely, together with data
+structures appropriate to the memory management needs.
+
+This interface is more tedious to use than ctypes' native interface, but
+it not meant for manual use. Instead, the translation toolchain maps
+uses of the usual ctypes interface in the RPython program to the more
+verbose but more regular rctypesobject interface.
+
+The rctypesobject library is mostly implemented as a family of functions
+that build and return new classes. For example, ``Primitive(Signed)``
+return the class corresponding to C objects of the low-level type
+``Signed`` (corresponding to ``long`` in C),
+``RPointer(Primitive(Signed))`` is the class corresponding to the type
+"pointer to long" and ``RVarArray(RPointer(Primitive(Signed)))`` return
+the class corresponding to arrays of pointers to longs.
+
+Each of these classes expose a similar interface: the ``allocate()``
+static method returns a new instance with its own memory storage
+(allocated with a separate ``lltype.malloc()`` as regular C memory, and
+automatically freed from the ``__del__()`` method of the instance). For
+example, ``Primitive(Signed).allocate()`` returns an instance that
+manages a word of C memory, large enough to contain just one ``long``.
+If ``x`` designates this instance, the expression ``pointer(x)``
+allocates and returns an instance of class
+``RPointer(Primitive(Signed))`` that manages a word of C memory, large
+enough to hold a pointer, and initialized to point to the previous
+``long`` in memory.
+
+The details of the interface can be found in the library's accompanying
+test file, `pypy/rlib/rctypes/test/test_rctypesobject.py`_.
+
+
+ControllerEntry
+---------------
+
+We implemented a generic way in the translation toolchain to map
+operations on arbitrary Python objects to calls to "glue" RPython
+classes and methods. This mechanism can be found in
+`pypy/rpython/controllerentry.py`_. In short, it allows code to
+register, for each full Python object, class, or metaclass, a
+corresponding "controller" class. During translation, the controller is
+invoked when the Python object, class or metaclass is encountered. The
+annotator delegates to the controller the meaning of all operations -
+instantiation, attribute reading and setting, etc. By default, this
+delegation is performed by replacing the operation by a call to a method
+of the controller; in this way, the controller can be written simply as
+a family of RPython methods with names like:
+
+* ``new()`` - called when the source RPython program tries to
+ instantiate the full Python class
+
+* ``get_xyz()`` and ``set_xyz()`` - called when the RPython program
+ tries to get or set the attribute ``xyz``
+
+* ``getitem()`` and ``setitem()`` - called when the RPython program uses
+ the ``[ ]`` notation to index the full Python object
+
+If necessary, the controller can also choose to entierely override the
+default annotating and rtyping behavior and insert its own. This is
+useful for cases where the method cannot be implemented in RPython,
+e.g. in the presence of a polymorphic operation that would cause the
+method signature to be ill-typed.
+
+For RCtypes, we implemented controllers that map the regular ctypes
+objects, classes and metaclasses to classes and operations from
+rctypesobject. This turned out to be a good way to separate the
+RCtypes implementation issues from the (sometimes complicated)
+interpretation of ctypes' rich and irregular interface.
+
+Performance
+-----------
+
+The greatest advantage of the ControllerEntry approach over the direct
+RTyping approach of the first RCtypes implementation is its higher
+level, giving flexibility. This is also potentially a disadvantage:
+there is for example no owner/alias analysis done during annotation;
+instead, all ctypes objects of a given type are implemented identically.
+We think that the general optimizations that we implemented - most
+importantly `malloc removal`_ - are either good enough to remove the
+overhead in the common case, or can be made good enough with some more
+efforts.
+
+XXX work in progress.
+
+
+------------------------------------------------------------
+ Appendix
+------------------------------------------------------------
.. _cpython-behavior:
-CPython's Ctype's Behaviour
-===========================
+Finest issues in the behavior of ctypes
+=======================================
For reference, this section describes ctypes behaviour on CPython, as
far as it is know to the author and relevant for rctypes. Of course,
@@ -400,7 +545,10 @@
.. _`ctypes documentation`: http://docs.python.org/dev/lib/module-ctypes.html
.. _ctypes: http://starship.python.net/crew/theller/ctypes/
+.. _libffi: http://sources.redhat.com/libffi/
.. _RPython: coding-guide.html#restricted-python
.. _`ctypes-0.9.9.6`: http://sourceforge.net/project/showfiles.php?group_id=71702&package_id=71318&release_id=411554
+.. _RTyping: rtyper.html
+.. _`malloc removal`: translation.html#malloc-removal
.. include:: _ref.txt
More information about the Pypy-commit
mailing list