[C++-sig] optimizing away calls to the python runtime -- was [detecting if a function is written in python or C++]

[François Duranleau]

On Tue, 17 Jan 2006, David Abrahams wrote:

> Mathieu Lacage <Mathieu.Lacage at sophia.inria.fr> writes:
>
>> My wrapped c++ code is not going to use boost::function but a similar
>> template-based thing which will end up generating classes such as
>> Callback1 in my example.
>
> Why reinvent the wheel?  boost::function will be more efficient,
> better tested, etc...

I have been following this discussion with great interest, since I am 
facing similar problems. I have some function templates to expose to 
Python and some of their template arguments are functions (functor or
function pointers), e.g. (dummy example):

template < typename Func >
void do_something( Func f )
{
     for ( int i = 0 ; i < 1000000 ; ++ i )
     {
         f( 10 ) ;
     }
}


The easiest way to expose it is, I think, something like this:

def( "do_something" , & do_something< object > ) ;


However, as pointed out by the timings given by Mathieu's last test, if 
the function parameter is to be called very often, it's a hit on 
performance. It could be possible to do something like the following, 
using boost::function :

typedef boost::function< void ( int ) > function_type ;

def( "do_something" , & do_something< function_type > ) ;


and then for each exposed functor, e.g.:

struct functor
{
     void operator () ( int a ) const
     {
         // ...
     }
} ;


you can use boost::python::implicitly_convertible:

class_< functor >( "Functor" )
     .def( "__call__" , & functor::operator () )
     ;

implicitly_convertible< functor , function_type >() ;


However, with this, you can't use Python callable objects anymore, unless 
we add an explicit converter to function_type for those:

function_type make_function( object o )
{
     return function_type( o ) ;
}


and then:

class_< function_type >( "Function" ) ;

def( "make_function" , & ::make_function ) ;


So in Python, you can do something like this:

// import the stuff

def func( a ) :
     # ...
     pass

do_something( Functor() )

do_something( make_function( func ) )


However, if we define a C++ function and want to pass it to do_something 
via Python, it seems like we have to either use make_function (and lose 
performance), or define a C++ wrapper functor. But in any case, the user 
has to explicitly use make_function for Python (actually, any other 
callable types not defined in your Boost.Python code) callable objects, 
and this non-uniformity between each type of call is not intuitive: a user 
must know if an object comes from Python or C++. Well, he could put 
make_function's everywhere, but he would lose the benefit. The ultimate 
goal would be to avoid any 'make_function'-like calls.

I put the complete example here (Makefile, test2.cc, test2.py):
http://www-etud.iro.umontreal.ca/~duranlef/python/

Actually, the whole point is, even using boost::function, it seems like it 
just moves the problem to someplace else, that is, how can you extract a 
C++ function/functor of unknown type (just the function signature is 
known) to store in the boost::function object, and otherwise (not a C++ 
object) just store the Python object, all that in a nice and easy way (for 
the user, at least)? Mathieu found a way, but it relies on some kind of 
hack with C++ name patterns. As a matter of fact, we could also use a hack 
with the __class__ attribute (it seems like the value for all Boost.Python 
classes is <type 'Boost.Python.class'>. But what about C++ functions? Are 
we doomed to have to deal with wrapper functors?

-- 
François Duranleau
LIGUM, Université de Montréal

"There are as many truths as there are people."
                                           - from _Neon Genesis Evangelion_