[C++-sig] high level overview of boost.python implementation
Martin Casado
casado2 at llnl.gov
Mon Jul 1 23:34:53 CEST 2002
Thanks for the comments Dave,
I'm going to redo some of the parts you commented on to clear things
up and resubmit.
~~m
> From: "Martin Casado" <casado2 at llnl.gov>
>
> > On Saturday 29 June 2002 10:58, you wrote:
> > > David Abrahams wrote:
> > > >Sounds great. I will try to throw you all a clue now and again also.
> >
> > I've found the most conceptually challenging aspects of Boost deal with
> > handeling the conversion between PyObjects and C++ arguments. In light
> > of this, I sat down and wrote a really rough overview of how things
> > seemed to work. I don't vouch for the correctness of this (it is most
>
> likely
>
> > wrong), but at the very least it might provide a basis for discussion. I
>
> am
>
> > really keen on getting a good set of overview documentation so... any
> > feedback,
> > additions, etc. would be greatly appreciated....
>
> This is great, Martin! Comments to follow...
>
> > File: HandlingArgumentsInBpl.txt
>
> ===========================================================================
> ====
>
> > Suggested Background Readings
> >
> > boost::function documentation:
>
> http://www.boost.org/libs/function/index.html
>
> > boost::bind documentation: http://www.boost.org/libs/bind/bind.html
>
> ===========================================================================
> ====
>
> > How Arguments From Python Are Handled
>
> ===========================================================================
> ====
>
> > It is perhaps easiest to start the discussion by describing the mechanism
> > used by BPL to convert python objects used as arguments from within
>
> python to
>
> > C++ objects for passing into wrapped functions.
> >
> > The main mechanism to handle the conversions is the struct
> >
> > template <typename T> struct arg_from_python
> >
> > arg_from_python is a class template which is used to convert
> > a PyObject* to a type T. arg_from_python has the following
> > members.
> >
> > arg_from_python(PyObject*) // constructor
> > bool convertible();
> > T operator()(PyObject*) const;
> >
> > convertible(..) returns true if BPL knows how to convert T
> > operator() handles the mechanics of the conversion. You could
> > use arg_from_python as follows to attempt to convert an arbitrary
> > PyObject into an object of type std::vector<int>
> >
> > std::vector<int> foo(PyObject* obj)
> > {
> > arg_from_python<std::vector<int> > from_py(obj);
> > if(from_py.convertible())
> > {
> > return from_py(obj);
> > }
> > }
>
> ...except that there's no return for the else case, and more-importantly,
> arg_from_python really shouldn't ever be touched by users. If they have to
> get their hands on a from_python converter, it should be
> converter::return_from_python<T>. The semantics for handling arguments and
> return types really are different, and return_from_python/arg_to_python
> match the users' needs and expectations much better than
> arg_from_python/to_python_value do.
>
> > This is simple enough, right? Unfortunately the implementation
> > of arg_from_python is not exactly straight-forward (for us
> > C++ weenies that is). Taking a peak at the declaration..
> > found in boost/python/arg_from_python.hpp
> >
> > template <class T>
> > struct arg_from_python
> >
> > : converter::select_arg_from_python<T>::type
> >
> > {
> > typedef typename converter::select_arg_from_python<T>::type
>
> base;
>
> > arg_from_python(PyObject*);
> > };
> >
> > we see that the actual implementations of convertible() and operator()
>
> are
>
> > dependant on some wiz-bang static mechanism for determining the
>
> correct
>
> > type via the class select_arg_from_python. The selection mechanism
>
> and
>
> > all implementations can be found in
> > boost/python/converter/arg_from_python.hpp.
> >
> > *** Note: discussion of registry here ?!?!
> > (I have a notion that the non specialized arg_from_python uses the
> > registry
> > to determine how to do the conversion using the registry)
>
> Yes, the non-specialized version uses converter functions which have been
> registered.
>
>
> ===========================================================================
> ====
>
> > How functions are called from python
> > (warning this is an incomplete and most
> > likely inaccurate description of the process)
>
> ===========================================================================
> ====
>
> > The last section discussed how, given an argument parameter such as
> > std::vector<int>, BPL handles the conversion, namely from creating an
>
> object
>
> > of
> > type arg_from_python<std::vector<int> > and using a specialized
>
> conversion
>
> > mechanism or using the default. (which probably uses the registry)
>
> However,
>
> > there has been no mention of how BPL actually gets access to the
>
> arguments of
>
> > the wrapped C++ method. The best start for a foray into this complex
>
> world is
>
> > to take a peak at boost/python/make_function.hpp. BPL functions are at
>
> the
>
> > highest level encapsulated in objects::function
> > (boost/python/objects/function.hpp) which handles overloading and
>
> ensuring
>
> > argument count is correct (and other things I don't know about).
>
> Actually that's about it. Well, there's method binding - so that when you
> access the function through a class instance it binds the function and a
> reference to the instance into a callable method object.
>
> > The creation of a default objects::function object as shown in
> > make_function.hpp is as follows:
> >
> > return new objects::function(
> > objects::py_function(
> >
> > ::boost::bind<PyObject*>(detail::caller(), f, _1,
>
> _2,
>
> > default_call_policies())),
> > detail::arg_tuple_size<F>::value);
> >
> > A function is created by a py_function object (which is really just a
> > typedef of boost:function2<PyObject*,PyObject*,PyObject*>) and an
>
> unisgned
>
> > int representing the minimum number of arguments accepted into the
> > function.
> > The minimum number of arguments is discovered through some static
>
> mechanism
>
> Right.
>
> > I'm not even going to begin to pretend to understand.
>
> Suit yerself. It's pretty simple <.0001 wink>
>
> > The py_function object is simply boost::function created from a bound
> > functor (detail::caller), which when called is passed f (the function
> > we are wrapping), the first two default arguments and a call policy
> > (in the case of the example it is passed the default call policy).
>
> Next
>
> > obvious next question is, "what does detail::caller do when called?"
> >
> > detail::caller's () operator actually invokes a static method in
> > detail::returning
> > (boost/python/detail/returning.hpp) as you can see in the following
>
> code
>
> > from boost/python/detail/caller.hpp.
> >
> > PyObject* operator()(
> > BOOST_PYTHON_FN(*f,0,args_)
> > , PyObject* args, PyObject* keywords
> > , P const& policies
> > ) const
> > {
> > return returning<R>::call(f, args, keywords,&policies);
> > }
> >
> >
> > returning<>::call does all sorts of fancy whizbang preprocessing and
> > metaprogramming tricks (feel free to browse the code to get a feel
>
> for
>
> > what I'm
> > talking about), which in essence boils down to the following.. (I
>
> think)
>
> > - For the 0th item in the tuple (self) check to see if it is
>
> convertible
>
> > to the correct type the method is being called on.
>
> That is of course different when the function pointer being wrapped is a
> regular function, instead of a member function.
>
> > /* check that each of the arguments is convertible */
> > /* self argument is special */
> > arg_from_python<A0&> c0(PyTuple_GET_ITEM(args_, 0));
> > if (!c0.convertible()) return 0;
> >
> > - For each item in the tuple (not including the self argument) check
>
> to
>
> > see
> > if that item is convertible to its respective argument in the C++
> > argument list.
> >
> > /* Unroll a loop for the rest of them */
> > BOOST_PP_REPEAT_FROM_TO(1,args,BOOST_PYTHON_ARG_CONVERTIBLE,nil)
> >
> > Where, BOOST_PYTHON_ARG_CONVERTIBLE is defined as ..
> >
> > # define BOOST_PYTHON_ARG_CONVERTIBLE(index,ignored)
> > arg_from_python<BOOST_PP_CAT(A,index)>
> > BOOST_PP_CAT(c,index)(PyTuple_GET_ITEM(args_, index));
> > if (!BOOST_PP_CAT(c,index).convertible()) return 0;
>
> It might be easier to look at
> boost/python/preprocessed/returning_non_void.hpp|returning_void.hpp, which
> have the preprocessor stuff completely expanded.
>
> > As the source code demonstrates, the mechanism for checking the
> > conversions
> > is instantiating an arg_from_python<arg> where arg is the argument
>
> type
>
> > and calling convertible() on that object.
> >
> > After the arguments are checked, the return value is checked to see
>
> if it
>
> > can be converted to a python object (a process not discussed in this
>
> text)
>
> > via the lines...
> >
> > /* find the result converter */
> > typedef typename P::result_converter result_converter;
> > typename mpl::apply1<result_converter,R>::type cr;
> > if (!cr.convertible()) return 0;
>
> Yes. Note that the availability of a result converter currently affects
> overload resolution. That's probably a (small) mistake.
>
> Nice work,
> Dave
>
>
>
>
> _______________________________________________
> C++-sig mailing list
> C++-sig at python.org
> http://mail.python.org/mailman/listinfo/c++-sig
More information about the Cplusplus-sig
mailing list