[C++-sig] high level overview of boost.python implementation

[Martin Casado]

On Saturday 29 June 2002 10:58, you wrote:
> David Abrahams wrote:
> >Sounds great. I will try to throw you all a clue now and again also.
> >

I've found the most conceptually challenging aspects of Boost deal with 
handeling the conversion between PyObjects and C++ arguments.  In light
of this, I sat down and wrote a really rough overview of how things
seemed to work.  I don't vouch for the correctness of this (it is most likely
wrong), but at the very least it might provide a basis for discussion.  I am
really keen on getting a good set of overview documentation so... any 
feedback,
additions, etc. would be greatly appreciated....

            ~~m


File:   HandlingArgumentsInBpl.txt
===============================================================================
 
Suggested Background Readings
 
boost::function documentation: http://www.boost.org/libs/function/index.html
boost::bind documentation:     http://www.boost.org/libs/bind/bind.html
 
===============================================================================
                    How Arguments From Python Are Handled
===============================================================================
 
It is perhaps easiest to start the discussion by describing the mechanism
used by BPL to convert python objects used as arguments from within python to
C++ objects for passing into wrapped functions.
 
  The main mechanism to handle the conversions is the struct
 
  template <typename T> struct arg_from_python
 
  arg_from_python is a class template which is used to convert
  a PyObject* to a type T.  arg_from_python has the following
  members.
 
      arg_from_python(PyObject*) // constructor
      bool convertible();
      T operator()(PyObject*) const;
 
   convertible(..) returns true if BPL knows how to convert T
   operator() handles the mechanics of the conversion.  You could
   use arg_from_python as follows to attempt to convert an arbitrary
   PyObject into an object of type std::vector<int>
 
   std::vector<int> foo(PyObject* obj)
   {
     arg_from_python<std::vector<int> > from_py(obj);
     if(from_py.convertible())
     {
        return from_py(obj);
     }
   }


   This is simple enough, right? Unfortunately the implementation
   of arg_from_python is not exactly straight-forward (for us
   C++ weenies that is).  Taking a peak at the declaration..
   found in boost/python/arg_from_python.hpp
 
        template <class T>
        struct arg_from_python
            : converter::select_arg_from_python<T>::type
        {
            typedef typename converter::select_arg_from_python<T>::type base;
            arg_from_python(PyObject*);
        };
 
   we see that the actual implementations of convertible() and operator() are
   dependant on some wiz-bang static mechanism for determining the correct
   type via the class select_arg_from_python.  The selection mechanism and
   all implementations can be found in 
boost/python/converter/arg_from_python.hpp.
 
   *** Note: discussion of registry here ?!?!
       (I have a notion that the non specialized arg_from_python uses the 
registry
        to determine how to do the conversion using the registry)
 
===============================================================================
             How functions are called from python
             (warning this is an incomplete and most
             likely inaccurate description of the process)
===============================================================================
 
   The last section discussed how, given an argument parameter such as
std::vector<int>, BPL handles the conversion, namely from creating an object 
of
type arg_from_python<std::vector<int> > and using a specialized conversion
mechanism or using the default. (which probably uses the registry)  However,
there has been no mention of how BPL actually gets access to the arguments of
the wrapped C++ method.  The best start for a foray into this complex world is
to take a peak at boost/python/make_function.hpp.  BPL functions are at the
highest level encapsulated in objects::function
(boost/python/objects/function.hpp) which handles overloading and ensuring
argument count is correct (and other things I don't know about).
 
   The creation of a default objects::function object as shown in
make_function.hpp is as follows:

  return new objects::function(
           objects::py_function(
                       ::boost::bind<PyObject*>(detail::caller(), f, _1, _2,
                       default_call_policies())),
                       detail::arg_tuple_size<F>::value);
 
   A function is created by a py_function object  (which is really just a
   typedef of boost:function2<PyObject*,PyObject*,PyObject*>) and an unisgned
   int representing the minimum number of arguments accepted into the 
function.
   The minimum number of arguments is discovered through some static mechanism
   I'm not even going to begin to pretend to understand.
 
   The py_function object is simply boost::function created from a bound
   functor (detail::caller), which when called is passed f (the function
   we are wrapping), the first two default arguments and a call policy
   (in the case of the example it is passed the default call policy).  Next
   obvious next question is, "what does detail::caller do when called?"
 
   detail::caller's () operator actually invokes a static method in 
detail::returning
   (boost/python/detail/returning.hpp) as you can see in the following code
   from boost/python/detail/caller.hpp.
 
    PyObject* operator()(
        BOOST_PYTHON_FN(*f,0,args_)
      , PyObject* args, PyObject* keywords
      , P const& policies
      ) const
    {
        return returning<R>::call(f, args, keywords,&policies);
    }


    returning<>::call does all sorts of fancy whizbang  preprocessing and
    metaprogramming tricks (feel free to browse the code to get a feel for 
what I'm
    talking about), which in essence boils down to the following.. (I think)
 
    - For the 0th item in the tuple (self) check to see if it is convertible
      to the correct type the method is being called on.
 
        /* check that each of the arguments is convertible */
        /* self argument is special */
        arg_from_python<A0&> c0(PyTuple_GET_ITEM(args_, 0));
        if (!c0.convertible()) return 0;
 
    - For each item in the tuple (not including the self argument) check to 
see
      if that item is convertible to its respective argument in the C++
      argument list.
 
       /* Unroll a loop for the rest of them */
       BOOST_PP_REPEAT_FROM_TO(1,args,BOOST_PYTHON_ARG_CONVERTIBLE,nil)
 
     Where, BOOST_PYTHON_ARG_CONVERTIBLE is defined as ..
 
    # define BOOST_PYTHON_ARG_CONVERTIBLE(index,ignored)
        arg_from_python<BOOST_PP_CAT(A,index)>
        BOOST_PP_CAT(c,index)(PyTuple_GET_ITEM(args_, index));
        if (!BOOST_PP_CAT(c,index).convertible()) return 0;

    As the source code demonstrates, the mechanism for checking the 
conversions
    is instantiating an arg_from_python<arg> where arg is the argument type
    and calling convertible() on that object.
 
    After the arguments are checked, the return value is checked to see if it
    can be converted to a python object (a process not discussed in this text)
    via the lines...
 
        /* find the result converter */
        typedef typename P::result_converter result_converter;
        typename mpl::apply1<result_converter,R>::type cr;
        if (!cr.convertible()) return 0;
 
    The call policy's precall() method is then called, the method is called 
with the
    converted arguments and finally the call policy's postcall method  is 
called and
    the result is returned.
 
      if (!policies->precall(args_)) return 0;
 
       PyObject* result = cr(
            ((BOOST_PYTHON_GET_ARG(0,nil)).*pmf)(
                BOOST_PP_ENUM_SHIFTED(args,BOOST_PYTHON_GET_ARG,nil))
            );
 
      return policies->postcall(args_, result);
 
   Of course for completeness it is worth discussing the actuall call to the 
method.

  Aka: ((BOOST_PYTHON_GET_ARG(0,nil)).*pmf)(
                BOOST_PP_ENUM_SHIFTED(args,BOOST_PYTHON_GET_ARG,nil))
 
   If the function pointer in question (aka pmf) is a pointer to a member 
function, it is
   bound to the first argument converted to the correct c++ type via 
BOOST_PYTHON_GET_ARG.
   This function is then called with the correct arguments which are 
converted also via
   BOOST_PYTHON_GET_ARG.  What does BOOST_PYTHON_GET_ARG do?  It calls the () 
operator
   on the arg_from_python type passing in, as an argument, python object from 
the argument
   tuple that is to be converted.
 
    # define BOOST_PYTHON_GET_ARG(index,ignored)
            BOOST_PP_CAT(c,index)(PyTuple_GET_ITEM(args_, index))
 
   Note, that there have been a number of things left out in this discussion.
   In reality pointers to member functions and pointers to non member 
functions
   are handled differently.  In the latter case, the object is passed in as 
the
   first argument to the function.  Also the content on the actual mechanics 
of
   the meta-programming has been virtually nil, mainly because I don't
   understand how it works :-P