[C++-sig] Marshalling Callbacks

[David Abrahams]

"William Trenker" <wtrenker at hotmail.com> writes:

> (This turned out to be a bit lengthy, but I have tried several
> approaches without success.  It is obvious that I am missing a key BPL
> concept (or two, or three).  I need to get this right because there
> are many places in FLTK that use this callback mechanism.)
>
> What I'm trying to figure out is the recommended / design-intended
> method for marshalling callbacks between the python layer and C++?
>
> 3-rd party GUI libraries, like FLTK, handle external events (such as
> mouse clicks) with a callback mechanism that is something like this
> very sketchy pseudo-code;
>
> -------------------3rd Party----------------------
> class Widget {     //low-level GUI object
> 	void callback(void* func, void* data) {

        void callback(void (*func)(void*), void* data)

??

> 	    //save callback address and data pointer
> 		cbfunc = func;
> 		cbdata = data;
> 	};
> 	void run() {
> 	    //wait for mouse clicks
> 		when (mouseClicked) (*cbaddr)(cbdata);
> private:
> 	void (*cbfunc)(void*);
> 	void* cbdata;  //opaque structure from GUI's perspective
> }
>
> Class Button : public Widget {
> public
> 	Widget();
> 	//NB: no callback() or run() defined here
> }
> ---------------------------------------------------
>
> An application, in C++, does something like this:
>
> Class A {
> public:
> 	A() { b = new Button(); };
> 	void onClick(void* appInfo) {
> 		//gets called by Widget::run
> 		cout << "Mouse was clicked!";
> 	};
> 	void execute(info) {
> 		b->callback(&onClick,&info);  //(1)
> 		b->run();                     //(2)
> 	}
> private:
> 	Button* b;
> 	whatever info;
> }
> At (1) the applicaton registers the callback member function so that,
> at (2), Widget::run knows what to call when the mouse is clicked.

This is a somewhat regressive approach. More modern systems would
use overridable virtual functions of a common base class that can
package arbitrary data with the function, and avoids the type-safety
problems associated with casting to/from void*:

    struct callback
    {
        virtual void operator()() = 0;
    };

Or much better still, boost::function, which has many other
advantages.

But I'm sure Bill S. thinks FLTK is F&L due to its use of raw function
and object pointers, so you're stuck with it.

> ---------------------------------------------------
>
> The python equivalent might be:
>
> import ext
> class A:
>    def __init__(self):
>       self.b = ext.Button()
>    def onClick(self,appInfo):
>       print "Mouse Clicked!"
>    def execute(self,info):
>       self.b.callback(self.onClick,info)
> 	  self.b.run()
> ---------------------------------------------------

Most likely the Python equivalent would also use nullary function
objects, which you could use lambda expressions user-defined function
objects to bind data into... but anyway, I'll try to avoid redesigning
your interface... ;-)

> When I initially set out,in blissful ignorance, to wrap the above API
> for Python, the following seemed to be the logical implementation:
>
> class_<Widget, boost::noncopyable>("Widget");
> class_<Button, bases<Widget>, boost::noncopyable>("Button")
>    .def("callback", &Widget::callback)
>    .def("run", &Widget::run);
>
> But, of course, the .def("callback"...) is not as simple as wrapping
> an int or a char*. It needs some additional options or features to
> marshall the callback mechanism so that a Python method (or function)
> gets saved and then eventually invoked by the 3rd-party C/C++ software.
>
> Where I'm stuck is what that marshalling mechanism looks like.  I'm
> aware of call<> and call_method<>; I've tinkered with handle<>'s and
> object's;  I've read the docs and studied the examples.
>
> Any advice or assistance is very greatly appreciated.

Clearly you've got a problem here.  Everything in Python, including
functions, are objects constructed at runtime.  A C++ function pointer
refers to code which must be determined entirely at compile time.

Q: What C++ function could you possibly write that would invoke
different Python functions at runtime, unless you also have some
additional data to indicate which Python function is to be invoked?

A: None.

So, there's no way to convert an arbitrary Python function argument
(which could itself be a Python function) into a void (*)(void*) which
captures the same meaning.

Fortunately, your callback interface has an additional void*
argument. So one thing you could do is to store a pointer to the
Python callable object in that void*.  Then you could write a thin
wrapper as follows:

    // helper function
    void call_object(void* py_function)
    {
        PyObject* o = (PyObject*)py_function;
        boost::python::call<void>(o);
    }

    // Wrap this one as a method of Widget; call it "callback":
    void callback_wrapper(Widget& w, boost::python::object f)
    {
        w.callback(call_object, f.ptr());
    } 

Notice that we now are back to having a nullary function interface
(the Python function takes no arguments), because we have no
additional arguments available to work with.  

We could do some things to get you back the ability to pass an
additional argument if you really want to, basically by writing a C++
binder, but even in doing so we'd run into your next big problem,
which is an issue even if we don't write a binder: the FLTK interface
doesn't give you any way to manage the lifetime of the callback data
pointer.  What if all the references to the Python callback function
disappear?  Well, then your data pointer will dangle.  This is why I
say it's a regressive interface.

You could cross your fingers and hope the function stays alive long
enough, but if you're wrong you'll crash and that's not very Pythonic,
is it?

You could increment the reference count on f.ptr() so that it will
leak.  Also not very Pythonic, but for most ordinary functions it
won't matter because they're very stable.  The problem is that since
it needs to carry its data, it's very unlikely to be an ordinary
function.

The problem with leaking is even worse with the "binder" solution
which preserves your unary function interface, since a new binder gets
created every time the callback() method is called, and they all have
to be leaked for sure.

So it looks like you have to choose the least odious alternative of
all of these.

FWIW, this problem has basically nothing to do with Boost.Python; it's
a problem of the FLTK interface.

-- 
                       David Abrahams
   dave at boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution