From pladow at gmail.com Wed Dec 17 05:13:32 2014 From: pladow at gmail.com (Peter LaDow) Date: Tue, 16 Dec 2014 20:13:32 -0800 Subject: [C++-sig] Passing opaque data to C++ callback from Python Message-ID: I'm embedding Python (using boost::python) into an application plugin that uses callbacks. Essentially, I want to do something like: In Python (say test.py): def do_something(): ... register_callback(do_something) And on the C++ side I register the register_callback() function: void register_callback(boost::python::object& o) { // Access some persistent state information } BOOST_PYTHON_MODULE(foo) { boost::python::def("register_callback", register_callback); } void library_entry() { PyImport_AppendInittab("foo", initfoo); PyInitialize(); // Create some persistent state information boost::python::exec_file("test.py", ...); } The issue here is that I need to create the Python context and store things away in an opaque pointer I return to the application. And when Python calls back into the C++, I need to recover that opaque value. For example: Application -- (calls) --> my C++ library -- (runs) --> Python script -- (calls) --> my C++ library For that last step, I need to recover some opaque data. Further, I need that opaque data to be persistent. The calls into my C++ library are callbacks from the application. The issue I'm having is trying to figure out how to pass that state information to the C++ register_callback() function. I've tried something like: namespace bp = boost::python; class state_info_t { }; void register_callback(std::shared_ptr& state, bp::object& o); { // Access some persistent state information } // Create some persistent state information std::shared_ptr state = std::make_shared(); PyImport_AppendInittab("foo", initfoo); Py_Initialize(); std::shared_ptr main_module = std::make_shared(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__")))); bp::object main_namespace = main_module->attr("__dict__"); std::shared_ptr foo_module = std::make_shared(bp::handle<>(PyImport_ImportModule("foo"))); main_namespace["foo"] = *foo_module; bp::scope foo_scope(*foo_module); // Both of these fail with _a lot_ of errors, most related to "no matching function call to 'get_signature' bp::def("register_callback", [&](bp::object& o) { register_callback(state, o); }, bp::arg("func")); bp::def("register_callback", std::bind(register_callback, state, std::placeholders::_1), bp::arg("func")); The other thought I had was to store the persistent data in the module's dictionary. But I don't know how to recover it in the callback. For example: // Create some persistent state information std::shared_ptr state = std::make_shared(); PyImport_AppendInittab("foo", initfoo); Py_Initialize(); std::shared_ptr main_module = std::make_shared(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__")))); bp::object main_namespace = main_module->attr("__dict__"); std::shared_ptr foo_module = std::make_shared(bp::handle<>(PyImport_ImportModule("foo"))); bp::object foo_namespace = main_module->attr("__dict__"); main_namespace["foo"] = *foo_module; foo_namespace["state"] = bp::handle<>(state); // Whatever the appropriate wrapper is Then in C++ side of register_callback: void register_callback(bp::object& o) { // How do I extract "state" from the context? } I'm not keen on this last one, since it exposes the state information to the script. I don't want to make the state information global since there may be multiple running Python instances. And regardless, with multiple Python instances I still need a way to determine which Python instance is running to select the appropriate state information. Any suggestions? From stefan at seefeld.name Thu Dec 18 18:16:39 2014 From: stefan at seefeld.name (Stefan Seefeld) Date: Thu, 18 Dec 2014 12:16:39 -0500 Subject: [C++-sig] Passing opaque data to C++ callback from Python In-Reply-To: References: Message-ID: <54930BF7.30401@seefeld.name> On 16/12/14 11:13 PM, Peter LaDow wrote: > I'm embedding Python (using boost::python) into an application plugin > that uses callbacks. Essentially, I want to do something like: > > In Python (say test.py): > > def do_something(): > ... > > register_callback(do_something) > > And on the C++ side I register the register_callback() function: > > void register_callback(boost::python::object& o) > { > // Access some persistent state information > } You defined 'do_something' as a callback, and registered it so it could be called from C++, yes ? Then, in your implementation of 'register_callback', 'o' is actually a reference to your callback function. Do you want to access the persistent state while registering the callback, or from within the callback ? Or am I misreading the code above ? Stefan > BOOST_PYTHON_MODULE(foo) > { > boost::python::def("register_callback", register_callback); > } > > void library_entry() > { > PyImport_AppendInittab("foo", initfoo); > > PyInitialize(); > > // Create some persistent state information > > boost::python::exec_file("test.py", ...); > } > > The issue here is that I need to create the Python context and store > things away in an opaque pointer I return to the application. And when > Python calls back into the C++, I need to recover that opaque value. > For example: > > Application -- (calls) --> my C++ library -- (runs) --> Python script > -- (calls) --> my C++ library > > For that last step, I need to recover some opaque data. Further, I > need that opaque data to be persistent. The calls into my C++ library > are callbacks from the application. The issue I'm having is trying to > figure out how to pass that state information to the C++ > register_callback() function. I've tried something like: > > namespace bp = boost::python; > > class state_info_t > { > }; > > void register_callback(std::shared_ptr& state, bp::object& o); > { > // Access some persistent state information > } > > // Create some persistent state information > std::shared_ptr state = std::make_shared(); > > PyImport_AppendInittab("foo", initfoo); > > Py_Initialize(); > > std::shared_ptr main_module = > std::make_shared(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__")))); > bp::object main_namespace = main_module->attr("__dict__"); > std::shared_ptr foo_module = > std::make_shared(bp::handle<>(PyImport_ImportModule("foo"))); > > main_namespace["foo"] = *foo_module; > > bp::scope foo_scope(*foo_module); > > // Both of these fail with _a lot_ of errors, most related to "no > matching function call to 'get_signature' > bp::def("register_callback", > [&](bp::object& o) { register_callback(state, o); }, > bp::arg("func")); > > bp::def("register_callback", > std::bind(register_callback, state, std::placeholders::_1), > bp::arg("func")); > > The other thought I had was to store the persistent data in the > module's dictionary. But I don't know how to recover it in the > callback. For example: > > // Create some persistent state information > std::shared_ptr state = std::make_shared(); > > PyImport_AppendInittab("foo", initfoo); > > Py_Initialize(); > > std::shared_ptr main_module = > std::make_shared(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__")))); > bp::object main_namespace = main_module->attr("__dict__"); > std::shared_ptr foo_module = > std::make_shared(bp::handle<>(PyImport_ImportModule("foo"))); > bp::object foo_namespace = main_module->attr("__dict__"); > > main_namespace["foo"] = *foo_module; > foo_namespace["state"] = bp::handle<>(state); // Whatever the > appropriate wrapper is > > Then in C++ side of register_callback: > > void register_callback(bp::object& o) > { > // How do I extract "state" from the context? > } > > I'm not keen on this last one, since it exposes the state information > to the script. > > I don't want to make the state information global since there may be > multiple running Python instances. And regardless, with multiple > Python instances I still need a way to determine which Python instance > is running to select the appropriate state information. > > Any suggestions? > _______________________________________________ > Cplusplus-sig mailing list > Cplusplus-sig at python.org > https://mail.python.org/mailman/listinfo/cplusplus-sig -- ...ich hab' noch einen Koffer in Berlin... From pladow at gmail.com Fri Dec 19 04:43:52 2014 From: pladow at gmail.com (Peter LaDow) Date: Thu, 18 Dec 2014 19:43:52 -0800 Subject: [C++-sig] Passing opaque data to C++ callback from Python In-Reply-To: <54930BF7.30401@seefeld.name> References: <54930BF7.30401@seefeld.name> Message-ID: Thanks for the reply! On Thu, Dec 18, 2014 at 9:16 AM, Stefan Seefeld wrote: > You defined 'do_something' as a callback, and registered it so it could > be called from C++, yes ? Then, in your implementation of Yes. At some pointer in the future, the Python function 'do_something' will be called from C++. > 'register_callback', 'o' is actually a reference to your callback Right. Using 'o' I can maintain a reference to the Python callback. > function. Do you want to access the persistent state while registering > the callback, or from within the callback ? Or am I misreading the code > above ? I need access to the persistent state while registering the callback. In fact, that persistent state is the context in which the callback from C++ into Python will be done. That is, I need to register 'o' (which references the Python 'do_something') in that persistent state. I think I figured out a mechanism. I was trying to make 'register_callback' a module function. I get that I could do something like: class persistent_state_t { public: ... void register_callback(boost::python::object& cb); }; Then I could register persistent_state_t::register_callback, then create an instance of persistent_state_t, and store that object. Then calls from Python to C++ would always be within the context of the particular instance of persistent_state_t. The issue was figuring out where to store an instance of persistent_state_t, make it visible to the user, and to prevent issues such as copies, deletion, etc. I wanted it to look like a module. The solution that I'm working with now is to store the instance of persistent_state_t in sys.modules with the name of the module. It is a class instance, rather than a module. But calls are within the context of the state. And since sys.modules (retrieved using PyImport_GetModuleDict) is a per-interpreter variable, I'm not polluting other interpreters running. Pete From stefan at seefeld.name Fri Dec 19 05:10:01 2014 From: stefan at seefeld.name (Stefan Seefeld) Date: Thu, 18 Dec 2014 23:10:01 -0500 Subject: [C++-sig] Passing opaque data to C++ callback from Python In-Reply-To: References: <54930BF7.30401@seefeld.name> Message-ID: <5493A519.6070205@seefeld.name> On 18/12/14 10:43 PM, Peter LaDow wrote: > Thanks for the reply! > > On Thu, Dec 18, 2014 at 9:16 AM, Stefan Seefeld wrote: >> You defined 'do_something' as a callback, and registered it so it could >> be called from C++, yes ? Then, in your implementation of > Yes. At some pointer in the future, the Python function > 'do_something' will be called from C++. > >> 'register_callback', 'o' is actually a reference to your callback > Right. Using 'o' I can maintain a reference to the Python callback. > >> function. Do you want to access the persistent state while registering >> the callback, or from within the callback ? Or am I misreading the code >> above ? > I need access to the persistent state while registering the callback. > In fact, that persistent state is the context in which the callback > from C++ into Python will be done. That sounds a bit confusing to me. :-) Typically, if you associate state with a callback (a "closure"), you think of it as an object whose member function is the callback. But here, the callback function lives in Python, while you want to associate state from the C++ runtime... > That is, I need to register 'o' > (which references the Python 'do_something') in that persistent state. > > I think I figured out a mechanism. I was trying to make > 'register_callback' a module function. I get that I could do > something like: > > class persistent_state_t > { > public: > ... > > void register_callback(boost::python::object& cb); > }; > > Then I could register persistent_state_t::register_callback, then > create an instance of persistent_state_t, and store that object. Then > calls from Python to C++ would always be within the context of the > particular instance of persistent_state_t. The issue was figuring out > where to store an instance of persistent_state_t, make it visible to > the user, and to prevent issues such as copies, deletion, etc. I > wanted it to look like a module. > > The solution that I'm working with now is to store the instance of > persistent_state_t in sys.modules with the name of the module. It is > a class instance, rather than a module. But calls are within the > context of the state. And since sys.modules (retrieved using > PyImport_GetModuleDict) is a per-interpreter variable, I'm not > polluting other interpreters running. OK, sounds like you have found what you need; great ! :-) Stefan -- ...ich hab' noch einen Koffer in Berlin... From pladow at gmail.com Fri Dec 19 05:26:46 2014 From: pladow at gmail.com (Peter LaDow) Date: Thu, 18 Dec 2014 20:26:46 -0800 Subject: [C++-sig] Passing opaque data to C++ callback from Python In-Reply-To: <5493A519.6070205@seefeld.name> References: <54930BF7.30401@seefeld.name> <5493A519.6070205@seefeld.name> Message-ID: On Thu, Dec 18, 2014 at 8:10 PM, Stefan Seefeld wrote: > That sounds a bit confusing to me. :-) > Typically, if you associate state with a callback (a "closure"), you > think of it as an object whose member function is the callback. But > here, the callback function lives in Python, while you want to associate > state from the C++ runtime... Exactly! There may be a better way to do what I'm trying to do. The relationship is one-to-many (i.e. a single Python object can be associated with many C++ objects). Because of this relationship, I need some context in which the Python object is called in order to make sure each of the C++ objects is aware that the callback occurred. That is, I don't want numerous C++ objects triggering a callback into Python. That shared information (whether or not the Python callback occurred) must be stored somewhere. Since I have no control over the order those C++ objects are invoked (they are passed to a 3rd party library), so when the first one is invoked, it needs to notify the others not to do the Python callback. Anyway, as you noted, I figured something out. I'm sure there's a more clever way to do this. But I'm an EE converted to the software world and I'm probably not thinking like a software engineer. Thanks, Pete From ilias at thechampion.ru Thu Dec 18 12:21:02 2014 From: ilias at thechampion.ru (ilias) Date: Thu, 18 Dec 2014 14:21:02 +0300 Subject: [C++-sig] How to override a property setter? Message-ID: <2250145.mxEssSEILL@kemmel> I got a class: class X { private: std::wstring text_; public: std::wstring text() const { return text_; } void set_text(const std::string& text) { text_ = from_utf8(text); } void set_text(const std::wstring& text) { text_ = text; } }; I'm trying to use it as a property: BOOST_PYTHON_MODULE(test) { class_("X") .add_property("text", &X::text, ) ; } What should I write to get following behavior: >>> x = X() >>> x.text = "abc" # I want X::set_text(const std::string&) to be invoked here >>> x.text = u"abc" # Here I expect X::set_text(const std::wstring&) to be called -- Ilya. From ilias at thechampion.ru Thu Dec 18 12:13:40 2014 From: ilias at thechampion.ru (ilias) Date: Thu, 18 Dec 2014 14:13:40 +0300 Subject: [C++-sig] Weird function call results Message-ID: <2422823.NDK6ldU9ec@kemmel> I got an overloaded function: void f(int n) { std::clog << "invoked f(int n = " << n << ")" << std::endl; } void f(double d) { std::clog << "invoked f(double d = " << d << ")" << std::endl; } If I put declarations in that order: void (*f_int)(int) = &f; void (*f_double)(double) = &f; BOOST_PYTHON_MODULE(test) { def("f", f_int); def("f", f_double); } No matter what I pass into the function f, f(double) is invoked every time. Even though I call f(int(1)) I will see "invoked f(double d = 1)". But if I declare the function f in reversed order: BOOST_PYTHON_MODULE(test) { def("f", f_double); def("f", f_int); } I will see "invoked f(int n = 5)" when I call f(5) and "invoked f(double d = 3.14)" when I call f(3.14) as it has to be. Why does it happen? Why does it depend on declaration order? -- Ilya. From stefan at seefeld.name Fri Dec 19 14:34:45 2014 From: stefan at seefeld.name (Stefan Seefeld) Date: Fri, 19 Dec 2014 08:34:45 -0500 Subject: [C++-sig] Weird function call results In-Reply-To: <2422823.NDK6ldU9ec@kemmel> References: <2422823.NDK6ldU9ec@kemmel> Message-ID: <54942975.1020402@seefeld.name> On 18/12/14 06:13 AM, ilias wrote: > I got an overloaded function: > > void f(int n) > { > std::clog << "invoked f(int n = " << n << ")" << std::endl; > } > > void f(double d) > { > std::clog << "invoked f(double d = " << d << ")" << std::endl; > } > > If I put declarations in that order: > > void (*f_int)(int) = &f; > void (*f_double)(double) = &f; > > BOOST_PYTHON_MODULE(test) > { > def("f", f_int); > def("f", f_double); > } > > No matter what I pass into the function f, f(double) is invoked every time. > Even though I call f(int(1)) I will see "invoked f(double d = 1)". > > But if I declare the function f in reversed order: > > BOOST_PYTHON_MODULE(test) > { > def("f", f_double); > def("f", f_int); > } > > I will see "invoked f(int n = 5)" when I call f(5) and "invoked f(double d = > 3.14)" when I call f(3.14) as it has to be. Why does it happen? Why does it > depend on declaration order? It definitely shouldn't. If it does, please submit a bug report including the test, as well as details as to what compiler and OS you were observing the error on. Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin... From amohr at pixar.com Sun Dec 21 05:30:29 2014 From: amohr at pixar.com (Alex Mohr) Date: Sat, 20 Dec 2014 20:30:29 -0800 Subject: [C++-sig] Weird function call results In-Reply-To: <54942975.1020402@seefeld.name> References: <2422823.NDK6ldU9ec@kemmel> <54942975.1020402@seefeld.name> Message-ID: <54964CE5.8000309@pixar.com> On 12/19/14 5:34 AM, Stefan Seefeld wrote: >> But if I declare the function f in reversed order: >> >> BOOST_PYTHON_MODULE(test) >> { >> def("f", f_double); >> def("f", f_int); >> } >> >> I will see "invoked f(int n = 5)" when I call f(5) and "invoked f(double d = >> 3.14)" when I call f(3.14) as it has to be. Why does it happen? Why does it >> depend on declaration order? > > It definitely shouldn't. If it does, please submit a bug report > including the test, as well as details as to what compiler and OS you > were observing the error on. It's not a bug, it's as designed. Boost.python tries function overloads in reverse registration order and picks the first one that works, in the sense that all the arguments convert. Alex From ilias at thechampion.ru Fri Dec 26 11:12:41 2014 From: ilias at thechampion.ru (ilias) Date: Fri, 26 Dec 2014 13:12:41 +0300 Subject: [C++-sig] Weird function call results In-Reply-To: <54964CE5.8000309@pixar.com> References: <2422823.NDK6ldU9ec@kemmel> <54942975.1020402@seefeld.name> <54964CE5.8000309@pixar.com> Message-ID: <12345722.uZtLC0P0Ic@kemmel> Alex, > It's not a bug, it's as designed. Boost.python tries function overloads > in reverse registration order and picks the first one that works, in the > sense that all the arguments convert. Is that behavior specified somewhere in the documentation? From amohr at pixar.com Sat Dec 27 20:45:33 2014 From: amohr at pixar.com (Alex Mohr) Date: Sat, 27 Dec 2014 11:45:33 -0800 Subject: [C++-sig] Weird function call results In-Reply-To: <12345722.uZtLC0P0Ic@kemmel> References: <2422823.NDK6ldU9ec@kemmel> <54942975.1020402@seefeld.name> <54964CE5.8000309@pixar.com> <12345722.uZtLC0P0Ic@kemmel> Message-ID: <549F0C5D.9090509@pixar.com> On 12/26/14 2:12 AM, ilias wrote: >> It's not a bug, it's as designed. Boost.python tries function overloads >> in reverse registration order and picks the first one that works, in the >> sense that all the arguments convert. > > Is that behavior specified somewhere in the documentation? I'm not sure it is. It ought to be. It is mentioned on the TODO page where it discusses changing this behavior. http://www.boost.org/doc/libs/1_57_0/libs/python/todo.html#best-match-overload-resolution If the behavior was changed it would surely break existing code. The current behavior is at least straightforward to reason about, and we've been able to get most cases to work (even tricky ones) by carefully arranging the registration order. For the rare case where a single order doesn't work, we simply write a lightweight wrapper that takes boost::python::object args and do the introspection and dispatch manually. Sometimes we have this generic overload as a "fallback" that we register first, letting boost.python handle the simple cases when it can. Alex