[pypy-commit] extradoc extradoc: in-progress

[arigo]

Author: Armin Rigo <arigo at tunes.org>
Branch: extradoc
Changeset: r5591:d5d6783367e1
Date: 2016-01-06 11:18 +0100
http://bitbucket.org/pypy/extradoc/changeset/d5d6783367e1/

Log:	in-progress

diff --git a/blog/draft/cffi-embedding.rst b/blog/draft/cffi-embedding.rst
--- a/blog/draft/cffi-embedding.rst
+++ b/blog/draft/cffi-embedding.rst
@@ -6,44 +6,55 @@
 Python programs, in a way that is both simple and that works across
 CPython 2.x and 3.x and PyPy.
 
-We are now adding support for *embedding* Python inside non-Python
-programs.  This is traditionally done using the CPython C API: from C
-code, you call ``Py_Initialize()`` and then some other functions like
+The major news of CFFI 1.4, released last december, was that you can
+now declare C functions with ``extern "Python"``, in the ``cdef()``.
+These magic keywords make the function callable from C (where it is
+defined automatically), but calling it will call some Python code
+(which you attach with the ``@ffi.def_extern()`` decorator).  This is
+useful because it gives a more straightforward, faster and
+libffi-independent way to write callbacks.  For more details, see `the
+documentation`_.
+
+You are, in effect, declaring a static family of C functions which
+call Python code.  The idea is to take pointers to them, and pass them
+around to other C functions, as callbacks.  However, the idea of a set
+of C functions which call Python code opens another path: *embedding*
+Python code inside non-Python programs.
+
+Embedding is traditionally done using the CPython C API: from C code,
+you call ``Py_Initialize()`` and then some other functions like
 ``PyRun_SimpleString()``.  In the simple cases it is, indeed, simple
-enough; but it can become a more complicated story if you throw in
-supporting application-dependent object types, and correctly running
-on multiple threads, and so on.
+enough; but it can become a complicated story if you throw in
+supporting application-dependent object types; and a messy story if
+you add correctly running on multiple threads, for example.
 
-Moreover, this approach is specific to CPython (2.x or 3.x, which you
-can do in a similar way).  It does not work on PyPy, which has its own
-smaller but very different `embedding API`_.
+Moreover, this approach is specific to CPython (2.x or 3.x).  It does
+not work at all on PyPy, which has its own very different, minimal
+`embedding API`_.
 
-The new-and-coming thing about CFFI, meant as replacement of the above
-solutions, is direct embedding support---and it does that with no
-fixed API at all.  The idea is to write some Python script with a
-``cdef()`` which declares a number of ``extern "Python"`` functions.
-When running the script, it creates the C source code and compiles it
-to a dynamically-linked library (``.so`` on Linux).  This is the same
-as in the regular API-mode usage, and ``extern "Python"`` was
-`introduced in CFFI 1.4`_.  What is new is that these ``extern
+The new-and-coming thing about CFFI 1.5, meant as replacement of the
+above solutions, is direct embedding support---with no fixed API at
+all.  The idea is to write some Python script with a ``cdef()`` which
+declares a number of ``extern "Python"`` functions.  When running the
+script, it creates the C source code and compiles it to a
+dynamically-linked library (``.so`` on Linux).  This is the same as in
+the regular API-mode usage.  What is new is that these ``extern
 "Python"`` can now also be *exported* from the ``.so``, in the C
 sense.  You also give a bit of initialization-time Python code
-directly in the script, which will be compiled into the ``.so``
-too.
+directly in the script, which will be compiled into the ``.so`` too.
 
-In other words, this library can now be used directly from any C
-program (and it is still importable in Python).  It exposes the C API
-of your choice, which you specified with the ``extern "Python"``
-declarations.  You can use it to make whatever custom API makes sense
-in your particular case.  You can even directly make a "plug-in" for
-any program that supports them, just by exporting the API expected for
-such plugins.
+This library can now be used directly from any C program (and it is
+still importable in Python).  It exposes the C API of your choice,
+which you specified with the ``extern "Python"`` declarations.  You
+can use it to make whatever custom API makes sense in your particular
+case.  You can even directly make a "plug-in" for any program that
+supports them, just by exporting the API expected for such plugins.
 
-This is still being finalized, but please try it out.  (You can also see
-`embedding.py`_ directly online for a quick glance.)  These are the
-instructions on Linux with CPython 2.7 (CPython 3.x and non-Linux
-platforms are still a work in progress right now, but this should be
-quickly fixed):
+This is still being finalized, but please try it out.  (You can also
+see `embedding.py`_ directly online for a quick glance.)  Here are
+below the instructions on Linux with CPython 2.7 (CPython 3.x and
+non-Linux platforms are still a work in progress right now, but this
+should be quickly fixed):
 
 * get the branch ``static-callback-embedding`` of CFFI::
 
@@ -59,7 +70,7 @@
       cd demo
       PYTHONPATH=.. python embedding.py
 
-* run ``gcc`` to build the C sources---on Linux::
+* this produces ``_embedding_cffi.c``; run ``gcc`` to build it---on Linux::
 
       gcc -shared -fPIC _embedding_cffi.c -o _embedding_cffi.so -lpython2.7 -I/usr/include/python2.7
 
@@ -75,22 +86,39 @@
 
 Very similar steps can be followed on PyPy, but it requires the
 ``cffi-static-callback-embedding`` branch of PyPy, which you must
-first translate from sources.
+first translate from sources.  The difference is only that you need to
+adapt the first ``gcc`` command line: replace ``-lpython2.7`` with
+``-lpypy-c`` and to fix the ``-I`` path (and possibly add a ``-L``
+path).
 
 Note that CPython/PyPy is automatically initialized (using locks in
 case of multi-threading) the first time any of the ``extern "Python"``
-functions is called from the C program.  At that time, the custom
-initialization-time Python code you put in
+functions is called from the C program.  (This should work even if two
+different threads call the first time a function from two *different*
+embedded CFFI extensions; in other words, explicit initialization is
+never needed).  The custom initialization-time Python code you put in
 ``ffi.embedding_init_code()`` is executed.  If this code starts to be
-big, you may consider moving it to independent modules or packages;
-then the initialization-time Python code only needs to import them
-(possibly after hacking around with ``sys.path``).
+big, you can move it to independent modules or packages.  Then the
+initialization-time Python code only needs to import them.  In that
+case, you have to carefully set up ``sys.path`` if the modules are not
+installed in the usual Python way.
+
+A better alternative would be to use virtualenv.  How to do that is
+not fully fleshed out so far.  You can certainly run the whole program
+with the environment variables set up by the virtualenv's ``activate``
+script first.  There are probably other solutions that involve using
+gcc's ``-Wl,-rpath=\$ORIGIN/`` or ``-Wl,-rpath=/fixed/path/`` options
+to load a specific libpython or libypypy-c library.  If you try it out
+and it doesn't work the way you would like, please complain :-)
 
 Another point: right now this does not support CPython's notion of
 multiple subinterpreters.  The logic creates a single global Python
-interpreter, and runs everything in that context.  Idea about how to
-support that cleanly would be welcome ``:-)`` More generally, any
-feedback is appreciated.
+interpreter, and runs everything in that context.  Maybe a future
+version would have an explicit API to do that---or maybe it should be
+the job of a 3rd-party extension module to provide a Python interface
+over the notion of subinterpreters...
+
+More generally, any feedback is appreciated.
 
 
 Have fun,