From seefeld at sympatico.ca Thu Apr 1 16:02:35 2004 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Thu, 01 Apr 2004 09:02:35 -0500 Subject: [C++-sig] missing symbol in boost_python 1.31 Message-ID: <124e50adac47035dc3a7ba312f206c31406c2486@Orthosoft.ca> hi there, I'v switched from boost.python 1.30.2 to 1.31 yesterday, and I'm getting a runtime error concerning a missing symbol boost::python::objects::function_object(boost::function2<_object*, _object*, _object*, std::allocator > const&, unsigned) I'm working with g++ 3.2 on a RH 8 system. Any ideas what could cause this ? I didn't make any changes to my code... Thanks, Stefan From seefeld at sympatico.ca Thu Apr 1 16:09:47 2004 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Thu, 01 Apr 2004 09:09:47 -0500 Subject: [C++-sig] missing symbol in boost_python 1.31 References: <124e50adac47035dc3a7ba312f206c31406c2486@Orthosoft.ca> Message-ID: <58506a0a221f939d2390086ed42233d8406c2637@Orthosoft.ca> sorry, that was a false alarm. A bug in my build system caused some object files not to be recompiled after the switch. It's working fine now. Thanks, Stefan From s_sourceforge at nedprod.com Thu Apr 1 12:39:01 2004 From: s_sourceforge at nedprod.com (Niall Douglas) Date: Thu, 01 Apr 2004 11:39:01 +0100 Subject: [C++-sig] Pyste and exporting inherited items Message-ID: <406BFF55.10019.12075479@localhost> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Bruno, are you aware that by adding the exporters.importing mechanism you have totally broken pyste's detection of class inheritance thus causing the class_<...bases > system to stop working? Furthermore nested class definitions get replicated, thus causing Boost.Python to fault on startup complaining about multiple definitions. The solution for others afflicted is to comment out line 55 "if not exporters.importing" in infos.py. Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQGvxR8EcvDLFGKbPEQKKxwCfRBndMv7ZIShBeodqkLHz/nToXPEAoK4U VoWQoNIhxixjreYHL6vmkUds =JYuC -----END PGP SIGNATURE----- From drostowsky at radioframenetworks.com Thu Apr 1 22:02:30 2004 From: drostowsky at radioframenetworks.com (David Rostowsky) Date: Thu, 1 Apr 2004 12:02:30 -0800 Subject: [C++-sig] Re: Virtual member functions cause exception in get_derived_class_object References: <001101c41734$5a143af0$64b3fea9@Presario2500> Message-ID: Attached are the files in question. Thanks for the direction. Im a newbie to these lists. In my original post I was trying to post the minimal that failed, so hence I wanted to give everything in the follow-up. Ill try to find a happier middle ground for future posts. "David Abrahams" wrote in message news:uwu4zvpit.fsf at boost-consulting.com... > > Hi David, > > First of all, please bring Boost.Python questions to the C++-sig: > http://www.boost.org/more/mailing_lists.htm#cplussig > > "David Rostowsky" writes: > > > I have a couple of really simple classes: > > > > class MyBase > > { > > public: > > int x; > > virtual int foo(int new_x) { }; // our derived classes may or may > > not override this function. Coders choice. > > }; > > > > class MyDerived: public MyBase > > { > > public: > > int y; > > MyDerived() { y = 0; }; > > int Get(void) { return y; }; > > void Set(int val) { y = val; }; > > }; > > > > My main goal in life is to create an instance of MyDerived class in > > C++: > > MyDerived my_derived = new MyDerived(); > > > > and then pass the pointer to a Python script where the Python script > > can call any of the public member functions. Yes, I am careful to keep > > the pointer around in the C++ code while the Python script uses it. > > So I expose the class (this may or may not be the source of my > > problems, need help): > > > > BOOST_PYTHON_MODULE(test) > > { > > using namespace boost::python; > > > > class_("MyDerived") > > .def("Get", &MyDerived::Get) > > .def("Set", &MyDerived::Set) > > ; > > } /* test*/ > > > > My impasse is that the virtual function seems to really give Boost > > hell when Im trying to pass the pointer to the class over to the > > python script. Heres the code snippet for that. > > Please post a complete, minimal, reproducible example. > > > Its the conversion thats just KILLING me. > > Chill, dude. You're still alive. At least someone's sending email > on your behalf ;-) > > > If I comment out the virtual function, everything works perfectly! I > > can pass the pointer over to my script, it calls the public member > > functions, and Im happy. HOWEVER, if I put the virtual function back > > in, the following line of code > > Well you've posted several lines there; can you be specific about > which one you're having problems with? > > > just EXPLODES! > > What does "EXPLODES" mean? Do you get a compilation error, a runtime > error, or something else? > > > // It crashes and burns in make_ptr_instance.cpp, function > > get_derived_class_object() > > boost::python::object my_arg(boost::ref(my_derived)); > > > > // pass the pointer over to the script now. > > int rval = callPython("myTestFunc", "Test String argument from c++", > > my_arg); > > -- > Dave Abrahams > Boost Consulting > www.boost-consulting.com > > _______________________________________________ > Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost > begin 666 test_script.py M=')Y. at T*(" @("!I;7!O71H;VXZ M($EM<&]R="!354-#145$140A(2$B#0IE>&-E<'0Z#0H@(" @('!R:6YT(")0 M>71H;VXZ($EM<&]R="!&04E,140N+BXB#0H-"B -"B,C(R,C(R,C(R,C(R,C M(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C M(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,-"B, at 5&5S="!M;V1U;&4-"B,C M(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C M(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,C(R,-"F1E9B!M M>51E71H;VXZ($ED96YT:71Y(&]F(&5X<&]S960 at 8W!P3V)J.B B M+ T*(" @("!C<'!/8FHN<')I;G10='(H*0T*(" @("!P71H;VXZ.FAA M;F1L93P^(&UY7VUO9'5L93L-"@T*8VQACL-"@E"87-E,2 at I('L@>CT@,3L@?3L-"@EV:7)T=6%L('9O:60@ M3VY+97E5<"AU;G-I9VYE9"!S:&]R="!V:7)T=6%L7VME>2D@>R!].PT*#0I] M.PT*#0IC;&%SPT*<')O=&5C=&5D M. at T*"6EN="!X.PT*#0IP=6)L:6,Z#0H-"@E497-T0R at I('L@>" ](# [('T[ M#0H)=F]I9"!3970H:6YT('9A;"D@>R!X(#T@=F%L.R!].PT*"6EN="!'970H M=F]I9"D@>R!R971U#L@?3L-"B @("!V;VED('!R:6YT4'1R*"D@>R!P MPT*(" @('5S:6YG(&YA;65S<&%C92!B;V]S=#HZ<'ET M:&]N.PT*#0H@("!O8FIE8W0@=&5S=$-L87-S(#T at 8VQA7!E71H;VXH71H;VXZ.F]B:F5C="8@;V)J M*3L-"@T*+R\@0V]N=F5N:65N8V4 at 9G5N8W1I;VYS#0IB;V]S=#HZ<'ET:&]N M.CIH86YD;&4\/B!I;7!OPT*"75S:6YG(&YA;65S<&%C92!B;V]S=#HZ<'ET:&]N.PT*#0H@ M(" @4'E/8FIE8W0@*FUO9'5L95]P='(@/2!0>4EM<&]R=%]);7!OPT*"0E/=71P=71$ M96)U9U-T71H;VXZ.F]B M:F5C="!G971?9G5N8W1I;VXH8V]N4]B:F5C=" J(&1I8W0@/2!0 M>4UO9'5L95]'971$:6-T*&UY7VUO9'5L92YG970H*2D[( T*"0T*(" @(&)O M;W-T.CIP>71H;VXZ.G-T71H;VXZ.F]B:F5C="AB;V]S=#HZ<'ET:&]N.CIH M86YD;&4\/BA0>41I8W1?1V5T271E;2AD:6-T+"!F=6YC3F%M92YP='(H*2DI M*3L-"@T*?2 O*B!G971?9G5N8W1I;VX@*B\-"@T*:6YT(&-A;&Q0>71H;VXH M8V]N71H;VXZ.F]B:F5C="!F=6YC M(#T at 9V5T7V9U;F-T:6]N*&-O;G-T7V-APT*:6YT(')V M86P[#0H-"E1E6]N92!I M2$@#0H)*B\@(" @(" @( T*(" @(" @("!O8FIE8W0 at 87)G,BAT M97-T,7!T51E Sorry if I am asking a question that has already been answered; the number of threads in this forum is formidable, and I can only see so far. This question was asking previously by my teammate Adrian Bentley; now that we know more about the problem, hopefully we can get an answer ;). I am working on a game engine that is attempting to use python for scripting support. And unusual requirement of our engine is the necessity of be able to manipulate types in C++ or Python, pretty much at the same time, e.g. dynamically instantiating an entity in C++ and then being able to manipulate it in python, and then updating its status in C++. I won't go into detail on why this is necessary, suffice to say, it has to deal with managing modules at run-time. I am most of the way to acheiving this goal; however, I am having trouble trying to access said C++ objects from python after I have instantiated them. to give you some idea what I am trying to do, following the boost tutorial I have a class that looks sorta like this --------------------------------------------------- struct PythonManager { dict main_namespace; handle<> main_module; //calls PyInitialize, and grabs the main module and namespace PythonManager(); //calls PyFinalize ~PythonManager(); //runs the interpreter on a single expression, and returns the result object EvaluateExpression(std::string &exp); //runs the interpreter on a multiline script void EvaluateScript(std::string &scr); struct PythonWrapper: public PyObject {}; PythonWrapper * MakeDualObject(int sizebytes, PyTypeObject *pto, std::string &varname) { //Get memory from Python and do necessary Python initialization PythonWrapper * mem = (PythonWrapper*)PyObject_Malloc(sizebytes); object obj( handle<>(PyObject_Init(mem, pto))); //put the variable into Python's variable namespace main_namespace[varname.c_str()] = obj; //set up some of my own tracking here.... return mem; } }; ----------------------------------------------------------- Now, the test that I am subjecting myself to see if all of this works is to have one of my defined types that I have extended into Python and another that I have created behind the scenes (like above) play well together. Unfortunately, I haven't gotten that far; so far, I am having trouble trying to get Python to recognize the types that I have just fed it. I successfully have extended a quaternion class in to python, and so I am using that as my test case, even stealing its typeobj * for use in the making these dual-language-objects. ------------------------------------------------------ struct PyQuat: public PyWrapper, public quat {}; //... _PythonManager.EvaluateScript("import mymath"); _PythonManager.EvaluateScript("x = mymath.quat(1,0,0,0)"); object testquat = _PythonManager.EvaluateExpression("x"); //verify that testquat really is a quat. PyWrapper *test = _PythonManager.MakeDualObject(sizeof(PyQuat), testquat.ptr()->objtype, "y"); //now the interpreter goes "boom!" _PythonManager.EvaluateScript("printquat(y)"); -------------------------------------------------------------- So, obviously what I am doing here is probably sub-optimal; still does anybody know of how to pull this off? Even better, if anybody knows of a more elegant way to pull this off, I am all ears. And again, once I pull that off above, I'll need to be able to make a behind-the-scenes-quat and made-in-python-quat play nicely; any advice on how to make that work would be really appreciated. Thanks for your time and patience, Mike Gonzales From dave at boost-consulting.com Thu Apr 1 22:45:55 2004 From: dave at boost-consulting.com (David Abrahams) Date: Thu, 01 Apr 2004 15:45:55 -0500 Subject: [C++-sig] Re: Virtual member functions cause exception in get_derived_class_object References: <001101c41734$5a143af0$64b3fea9@Presario2500> Message-ID: "David Rostowsky" writes: > Attached are the files in question. Thanks for the direction. Im a newbie to > these lists. In my original post I was trying to post the minimal that > failed, so hence I wanted to give everything in the follow-up. Ill try to > find a happier middle ground for future posts. Also, it'd be good to post instructions for running the test; e.g. where is the test script placed? Incidentally, the slightly modified program I enclosed produces: C++ : Initializing Python and loading script 'import site' failed; use -v for traceback Python: Import SUCCEEDED!!! C++ : Ptr=0x00867958 C++ : test1ptr->Get() in C++ before Python call: 0 Python: msg = Test String argument from c++ Python: Identity of 'local' MyClass object: C++ : Ptr=0x00868B70 Python: Current state: 0 Python: New state : 1234 Python: Identity of exposed cppObj: C++ : Ptr=0x00867958 Python: cppObj.Get() on entry: 0 Python: cppObj.Get() on exit : 9 C++ : callPython() returned: 999 C++ : test1ptr->Get() after Python call: 9 C++ : Press any key to quit -------------- next part -------------- A non-text attachment was scrubbed... Name: test.cpp Type: text/x-cplusplus Size: 6628 bytes Desc: not available URL: -------------- next part -------------- -- Dave Abrahams Boost Consulting www.boost-consulting.com From dave at boost-consulting.com Thu Apr 1 22:59:06 2004 From: dave at boost-consulting.com (David Abrahams) Date: Thu, 01 Apr 2004 15:59:06 -0500 Subject: [C++-sig] Re: Accessing instances of C++ objects in Python - Plus more References: <3.0.6.32.20040401122040.009e8458@mail.digipen.edu> Message-ID: Mike Gonzales writes: > Sorry if I am asking a question that has already been answered; the number > of threads in this forum is formidable, and I can only see so far. This > question was asking previously by my teammate Adrian Bentley; now that we > know more about the problem, hopefully we can get an answer ;). > > I am working on a game engine that is attempting to use python for > scripting support. And unusual requirement of our engine is the necessity > of be able to manipulate types in C++ or Python, pretty much at the same > time, e.g. dynamically instantiating an entity in C++ and then being able > to manipulate it in python, and then updating its status in C++. I won't > go into detail on why this is necessary, No problem; that's what Boost.Python is for. > suffice to say, it has to deal with managing modules at run-time. > I am most of the way to acheiving this goal; however, I am having trouble > trying to access said C++ objects from python after I have instantiated them. > > to give you some idea what I am trying to do, following the boost tutorial > I have a class that looks sorta like this > > --------------------------------------------------- > struct PythonManager > { > dict main_namespace; > handle<> main_module; > > //calls PyInitialize, and grabs the main module and namespace > PythonManager(); > > //calls PyFinalize > ~PythonManager(); PyFinalize isn't supported with Boost.Python. See http://www.boost-consulting.com/boost/libs/python/todo.html#pyfinalize-safety > //runs the interpreter on a single expression, and returns the result > object EvaluateExpression(std::string &exp); > //runs the interpreter on a multiline script > void EvaluateScript(std::string &scr); > > struct PythonWrapper: public PyObject > {}; > > PythonWrapper * MakeDualObject(int sizebytes, PyTypeObject *pto, > std::string &varname) > { > //Get memory from Python and do necessary Python initialization > PythonWrapper * mem = (PythonWrapper*)PyObject_Malloc(sizebytes); > object obj( handle<>(PyObject_Init(mem, pto))); If pto refers to a wrapped C++ class, this is the wrong way to go about it. > //put the variable into Python's variable namespace > main_namespace[varname.c_str()] = obj; > > //set up some of my own tracking here.... > > return mem; > } > }; > ----------------------------------------------------------- > > Now, the test that I am subjecting myself to see if all of this works is to > have one of my defined types that I have extended into Python and another > that I have created behind the scenes (like above) play well together. > Unfortunately, I haven't gotten that far; so far, I am having trouble > trying to get Python to recognize the types that I have just fed it. > > I successfully have extended a quaternion class in to python, and so I am > using that as my test case, even stealing its typeobj * for use in the > making these dual-language-objects. > > ------------------------------------------------------ > struct PyQuat: public PyWrapper, public quat > {}; Is this PyQuat thing wrapped with class_ ?? > //... > _PythonManager.EvaluateScript("import mymath"); > _PythonManager.EvaluateScript("x = mymath.quat(1,0,0,0)"); > object testquat = _PythonManager.EvaluateExpression("x"); > > //verify that testquat really is a quat. > > PyWrapper *test = _PythonManager.MakeDualObject(sizeof(PyQuat), > testquat.ptr()->objtype, "y"); > //now the interpreter goes "boom!" > _PythonManager.EvaluateScript("printquat(y)"); > -------------------------------------------------------------- > > So, obviously what I am doing here is probably sub-optimal; still does > anybody know of how to pull this off? It's not even clear what you're trying to pull off. Incomplete code snippets don't leave us with much of a clue as to what the problem might be. The best way to get help is to reduce your problem to a complete, minimal example. I'm not sure if this addresses your problem, but you can capture any Python wrapper for a C++ class this way object T_class = class_("T") // methods, etc... ; and then you can make a wrapped T object this way: object wrapped_T = T_class(/*whatever args you like*/); and you can access the underlying T this way: T& t = extract(wrapped_T); > Even better, if anybody knows of a more elegant way to pull this > off, I am all ears. And again, once I pull that off above, I'll > need to be able to make a behind-the-scenes-quat and > made-in-python-quat play nicely; any advice on how to make that work > would be really appreciated. Sorry, most of your questions are too imprecise to answer with confidence. What does "play nicely" mean? -- Dave Abrahams Boost Consulting www.boost-consulting.com From dave at boost-consulting.com Thu Apr 1 23:02:26 2004 From: dave at boost-consulting.com (David Abrahams) Date: Thu, 01 Apr 2004 16:02:26 -0500 Subject: [C++-sig] Re: Virtual member functions cause exception in get_derived_class_object References: <001101c41734$5a143af0$64b3fea9@Presario2500> Message-ID: Please see my replies on the C++-sig. Cheers! -- Dave Abrahams Boost Consulting www.boost-consulting.com From halbert at bbn.com Thu Apr 1 23:32:02 2004 From: halbert at bbn.com (Dan Halbert) Date: Thu, 1 Apr 2004 16:32:02 -0500 Subject: [C++-sig] Pyste: Virtual class wrapper doesn't refcount object Message-ID: <200404012132.i31LW2Z12116@rolex.bbn.com> In Pyste from the 1.31.0 release: If Pyste finds any virtual methods in a class it is wrapping (if I understand correctly), it generates a class wrapper for that class that looks something like this: struct VMClass_Wrapper : VMClass_Wrapper { (PyObject* self_): VMClass(), self(self_) {} void f(long int p0) { call_method< void >(self, "f", p0); } PyObject* self; }; Note that the self object is not properly reference-counted. It seems to me it should be, since it's held by the struct, and can disappear from scope in the Python code, and so would go out of existence. I had just such an error when using this class in a callback situation: registerMyCallback(VMClass(), ...) # uses an anonymous VMClass() object I fixed this by modifying the code generated by Pyste to use handle<> and borrowed(): struct VMClass_Wrapper : VMClass_Wrapper { (PyObject* self_): VMClass(), self(borrowed(self_)) {} // was self(self_) void f(long int p0) { call_method< void >(self.get(), "f", p0); // was just self } handle<> self; // was PyObject* self; }; Anything wrong with this? I believe I am using borrowed() properly here, but I'm not 100% sure. If it's OK, it's a fix to make to Pyste. Dan From dave at boost-consulting.com Fri Apr 2 00:20:52 2004 From: dave at boost-consulting.com (David Abrahams) Date: Thu, 01 Apr 2004 17:20:52 -0500 Subject: [C++-sig] Re: Pyste: Virtual class wrapper doesn't refcount object References: <200404012132.i31LW2Z12116@rolex.bbn.com> Message-ID: Dan Halbert writes: > In Pyste from the 1.31.0 release: > > If Pyste finds any virtual methods in a class it is wrapping (if I > understand correctly), it generates a class wrapper for that class > that looks something like this: > > > struct VMClass_Wrapper : VMClass_Wrapper > { > (PyObject* self_): > VMClass(), self(self_) {} > > void f(long int p0) { > call_method< void >(self, "f", p0); > } > > PyObject* self; > }; > > > Note that the self object is not properly reference-counted. It seems > to me it should be No, Pyste generates correct code for this case. See http://www.boost.org/libs/python/doc/tutorial/doc/class_virtual_functions.html. The Python class already owns the C++ class. That would create an ownership cycle. > since it's held by the struct, and can disappear from scope in the > Python code, and so would go out of existence. > > I had just such an error when using this class in a callback situation: > > registerMyCallback(VMClass(), ...) # uses an anonymous VMClass() object I'm not sure what that's doing, but your problem lies elsewhere. -- Dave Abrahams Boost Consulting www.boost-consulting.com From dave at boost-consulting.com Thu Apr 1 22:34:20 2004 From: dave at boost-consulting.com (David Abrahams) Date: Thu, 01 Apr 2004 15:34:20 -0500 Subject: [C++-sig] Re: Virtual member functions cause exception in get_derived_class_object References: <001101c41734$5a143af0$64b3fea9@Presario2500> Message-ID: "David Rostowsky" writes: > Attached are the files in question. Thanks for the direction. Im a newbie to > these lists. In my original post I was trying to post the minimal that > failed, so hence I wanted to give everything in the follow-up. Ill try to > find a happier middle ground for future posts. Can you try to answer the other questions I asked in my reply? -- Dave Abrahams Boost Consulting www.boost-consulting.com From aashish at vrac.iastate.edu Fri Apr 2 04:59:56 2004 From: aashish at vrac.iastate.edu (aashish) Date: Thu, 1 Apr 2004 20:59:56 -0600 Subject: [C++-sig] Re: conversion of python dictionary to shared_ptrs In-Reply-To: <93618711046370@webmail.iastate.edu> Message-ID: <200404020301.i3231v3N5674255@vracs001.vrac.iastate.edu> Hi, I need to pass a dictionary from python to C++ function which is defined as void setVariables(ValueMapPtr variables); where I have typedef ValueMapPtr as //a shared ptr of a value map typedef boost::shared_ptr< ValueMap > ValueMapPtr; // And typedef std::map< std::string, std::vector > ValueMap; Is it possible?? Do I need to use register_ptr_to_python ? Any help is highly appreciated. Thanks, Aashish From halbert at jubilee.bbn.com Fri Apr 2 06:02:06 2004 From: halbert at jubilee.bbn.com (Dan Halbert) Date: Thu, 1 Apr 2004 23:02:06 -0500 Subject: [C++-sig] Re: Pyste: Virtual class wrapper doesn't refcount object In-Reply-To: (message from David Abrahams on Thu, 01 Apr 2004 17:20:52 -0500) References: <200404012132.i31LW2Z12116@rolex.bbn.com> Message-ID: <200404020402.i32426C03669@hbo.bbn.com> >> Note that the self object is not properly reference-counted. It seems >> to me it should be > >No, Pyste generates correct code for this case. See >http://www.boost.org/libs/python/doc/tutorial/doc/class_virtual_functions.html. OK, I see I am incorrectly blaming Pyste. >The Python class already owns the C++ class. That would create an >ownership cycle. > >> since it's held by the struct, and can disappear from scope in the >> Python code, and so would go out of existence. OK, below is a complete tested small BPL example of the problem I had before I added the borrowed and handle<> in the callback wrapper. I must be misunderstanding something about object ownership. ============================================================================== #include #include using namespace boost::python; // Original C++ class struct Callback { virtual void do_callback() = 0; }; // BPL-tutorial-style wrapper, same as generated by Pyste struct Callback_Wrapper : Callback { Callback_Wrapper(PyObject* self_) : Callback(), self(self_) {} void do_callback() { call_method< void >(self, "do_callback"); } PyObject* self; }; Callback* registered_callback; void register_callback(Callback* cb) { registered_callback = cb; } void call_callback() { registered_callback->do_callback(); } BOOST_PYTHON_MODULE(cb) { class_("Callback", init< >()) .def("do_callback", pure_virtual(&Callback::do_callback)); def("register_callback", register_callback); def("call_callback", call_callback); } ============================================================================== ============================================================================== Now, I'll try it in Python: % python Python 2.3.3 (#1, Mar 1 2004, 12:51:08) [GCC 3.2.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import cb >>> class MyCallback(cb.Callback): # subclass the C++ class ... def do_callback(self): ... print "callback called" ... >>> cback = MyCallback() # create an instance & remember it >>> cb.register_callback(cback) # register the callback >>> cb.call_callback() # call the callback callback called # callback works! >>> cb.register_callback(MyCallback()) # anonymous instance, short lifetime >>> cb.call_callback() # call the new callback instance Segmentation fault # BOOM! ============================================================================== If I wrote register_callback(MyCPPCallback()); // BAD IDEA; Callback doesn't live long in C++, I wouldn't expect that to work at all, because the MyCPPCallback() object doesn't live past its use as a parameter. Maybe I shouldn't expect it to work in Python either? I want the C++ code to take ownership of the MyCallback PyObject, or more simply, just increment its reference count. Perhaps I need a policy for the passed-in MyCallback object?? The Call Policies part of the tutorial seems applicable. But all the appropriate-sounding policies seem to apply only to return values. What am I doing wrong? Thanks, Dan From danielhorn at mindspring.com Fri Apr 2 10:40:05 2004 From: danielhorn at mindspring.com (Daniel Horn) Date: 02 Apr 2004 00:40:05 -0800 Subject: [C++-sig] Vega Strike uses boost::python Message-ID: <1080895205.29404.7.camel@mirror> Vega Strike is the 3D Space Simulator that allows you to trade and bounty hunt in a vast universe. Players face dangers, decisions, piracy, and aliens... Instead of going the hackneyed route of using Lua for game scripting, Vega Strike (http://vegastrike.sourceforge.net/) has decided to base its scripting on python, using boost as the layer between the class hierarchy in python and the class hierarchy in C++. The result is a very flexible scripting system that treats units as native python classes when designing missions or writing AI's. A large economic and planetary simulation is currently being run in the background in python and the results are returned back into C++ in the form of various factions' spaceships appearing near worlds that they are simulated to be near in python if the player is in the general neighborhood. From dave at boost-consulting.com Fri Apr 2 15:06:37 2004 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 02 Apr 2004 08:06:37 -0500 Subject: [C++-sig] Re: Pyste: Virtual class wrapper doesn't refcount object References: <200404012132.i31LW2Z12116@rolex.bbn.com> <200404020402.i32426C03669@hbo.bbn.com> Message-ID: Dan Halbert writes: > >> Note that the self object is not properly reference-counted. It seems > >> to me it should be > > > >No, Pyste generates correct code for this case. See > >http://www.boost.org/libs/python/doc/tutorial/doc/class_virtual_functions.html. > > OK, I see I am incorrectly blaming Pyste. > > >The Python class already owns the C++ class. That would create an > >ownership cycle. > > > >> since it's held by the struct, and can disappear from scope in the > >> Python code, and so would go out of existence. > > > OK, below is a complete tested small BPL example of the problem I had > before I added the borrowed and handle<> in the callback wrapper. I > must be misunderstanding something about object ownership. Yep. The basic principle is: A wrapped C++ object is always owned by the Python object that wraps it. Another principle you can use is: A when a Python object x is converted to a shared_ptr p, p maintains a reference to x, not just to *p. I'm inverting the order of your code snippets: | ============================================================================== | | Now, I'll try it in Python: | | | % python | Python 2.3.3 (#1, Mar 1 2004, 12:51:08) | [GCC 3.2.2] on linux2 | Type "help", "copyright", "credits" or "license" for more information. |>>> import cb |>>> class MyCallback(cb.Callback): # subclass the C++ class | ... def do_callback(self): | ... print "callback called" | ... |>>> cback = MyCallback() # create an instance & remember it |>>> cb.register_callback(cback) # register the callback |>>> cb.call_callback() # call the callback | callback called # callback works! Right; the callback's lifetime is maintained by the the module dict's 'cback' entry. |>>> cb.register_callback(MyCallback()) # anonymous instance, short lifetime |>>> cb.call_callback() # call the new callback instance | Segmentation fault # BOOM! Right; the Python object is gone at this point. | ============================================================================== | If I wrote | | register_callback(MyCPPCallback()); // BAD IDEA; Callback doesn't live long Right. I don't know what MyCPPCallback is, but if it's a class, you've got the same problem as above. | in C++, I wouldn't expect that to work at all, because the MyCPPCallback() | object doesn't live past its use as a parameter. Maybe I shouldn't | expect it to work in Python either? Not unless you do something to manage the lifetime of the Python object. | I want the C++ code to take ownership of the MyCallback PyObject Exactly. | or more simply, just increment its reference count. No, you don't want that. In that case it'll leak. | Perhaps I need a policy for the passed-in MyCallback object?? I don't see how it could help. | The Call Policies part of the tutorial seems applicable. But all the | appropriate-sounding policies seem to apply only to return values. | What am I doing wrong? The solution is to get the C++ code to hang onto the Python object, rather than just the C++ object. Here's one approach: | | ============================================================================== | #include | #include #include | using namespace boost::python; | | // Original C++ class | struct Callback | { | virtual void do_callback() = 0; | }; | | // BPL-tutorial-style wrapper, same as generated by Pyste | struct Callback_Wrapper : Callback | { | Callback_Wrapper(PyObject* self_) : Callback(), self(self_) {} | void do_callback() { call_method< void >(self, "do_callback"); } | PyObject* self; | }; typedef boost::shared_ptr CallbackPtr; | // Callback* registered_callback; CallbackPtr registered_callback; | // void register_callback(Callback* cb) void register_callback(CallbackPtr cb) | { | registered_callback = cb; | } | | void call_callback() | { | registered_callback->do_callback(); | } | | BOOST_PYTHON_MODULE(cb) | { | class_("Callback", init< >()) | .def("do_callback", pure_virtual(&Callback::do_callback)); | | def("register_callback", register_callback); | def("call_callback", call_callback); | } | ============================================================================== HTH, -- Dave Abrahams Boost Consulting www.boost-consulting.com From dave at boost-consulting.com Fri Apr 2 15:13:28 2004 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 02 Apr 2004 08:13:28 -0500 Subject: [C++-sig] Re: Vega Strike uses boost::python References: <1080895205.29404.7.camel@mirror> Message-ID: Thanks, Daniel! I added this to the projects page. I hope you don't mind, but... Daniel Horn writes: > Vega Strike is the 3D Space Simulator that allows you to trade and > bounty hunt in a vast universe. Players face dangers, decisions, piracy, > and aliens... > > Instead of going the hackneyed route of using Lua for game scripting, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...I left the above phrase out. Publicly bashing Lua would be unseemly for Boost.Python ;-) > Vega Strike (http://vegastrike.sourceforge.net/) > has decided to base its scripting on python, using boost as the layer > between the class hierarchy in python and the class hierarchy in C++. > The result is a very flexible scripting system that treats units as > native python classes when designing missions or writing AI's. > > A large economic and planetary simulation is currently being run in the > background in python and the results are returned back into C++ in the > form of various factions' spaceships appearing near worlds that they are > simulated to be near in python if the player is in the general > neighborhood. -- Dave Abrahams Boost Consulting www.boost-consulting.com From mgonzale at digipen.edu Fri Apr 2 19:34:24 2004 From: mgonzale at digipen.edu (Mike Gonzales) Date: Fri, 02 Apr 2004 09:34:24 -0800 Subject: [C++-sig] Re: Accessing instances of C++ objects in Python - Plus more Message-ID: <3.0.6.32.20040402093424.009e8458@mail.digipen.edu> Thank you for your input - I am incorporating it to make my code neater and more precise. Anyway, I guess my question devolves to this: How do I access a C++ variable from python, and what steps do I need to take? The most obvious hack that I thought of was to try and manually insert the python-wrapped C++ variable into the main module's namespace dictionary; Is there a better way than that? Thanks for your time and patience, Mike From s_sourceforge at nedprod.com Fri Apr 2 18:55:27 2004 From: s_sourceforge at nedprod.com (Niall Douglas) Date: Fri, 02 Apr 2004 17:55:27 +0100 Subject: [C++-sig] Loading Boost.Python made bindings is horribly slow? Message-ID: <406DA90F.23013.18865357@localhost> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 With GCC v3.4 from CVS, I finally have build time down to only nine hours now rather than the 20+ it was with GCC v3.3. However, when loading the bindings into python on Linux v2.4 kernel it takes well over six minutes. Is anyone else having this problem? I don't think it's BPL's fault really as on Windows it only takes a second or two so I'm guessing it's the dynamic linker. I'd imagine that that prebinding linkage feature of newer Linuces will fix this problem - in the meantime, any suggestions? (Here's a thought - GCC outputs far too many symbols into the public ELF table - if GCC had a dllexport/dllimport mechanism like on Windows, the amount of symbols needing to be linked could be reduced very significantly indeed. As it so happens, someone submitted a patch to enable per-class symbol visibility specifications using the same syntax as MSVC, it just needs extending to use a command line argument specified default so you can mark everything default hidden. If people here think this a good idea, I may try tackling it). Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQG2bA8EcvDLFGKbPEQKACQCffwTwb5rzU0ikzQ3aWATWRKHirmEAoJez fuxMMJ25M6rhVO6No/2o4OKS =CyW3 -----END PGP SIGNATURE----- From seefeld at sympatico.ca Fri Apr 2 20:10:21 2004 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Fri, 02 Apr 2004 13:10:21 -0500 Subject: [C++-sig] Loading Boost.Python made bindings is horribly slow? References: <406DA90F.23013.18865357@localhost> Message-ID: <0a25731b1342271ceab98f36271d7dd8406db042@Orthosoft.ca> Niall Douglas wrote: > Is anyone else having this problem? I don't think it's BPL's fault > really as on Windows it only takes a second or two so I'm guessing > it's the dynamic linker. I'd imagine that that prebinding linkage > feature of newer Linuces will fix this problem - in the meantime, any > suggestions? We experience the same: in our case it's already the linking of the module that takes a long time because we split up the code into multiple translation units. > (Here's a thought - GCC outputs far too many symbols into the public > ELF table - if GCC had a dllexport/dllimport mechanism like on > Windows, the amount of symbols needing to be linked could be reduced > very significantly indeed. As it so happens, someone submitted a The simplest thing to do (in this particular context) is to put everything into an anonymous namespace. Everything but the actual python module init function ('BOOST_PYTHON_MODULE()'). That should restrict the number of the exposed symbols to a strict minimum. Stefan From jdbrandm at unity.ncsu.edu Fri Apr 2 21:06:18 2004 From: jdbrandm at unity.ncsu.edu (Jonathan Brandmeyer) Date: Fri, 02 Apr 2004 14:06:18 -0500 Subject: [C++-sig] Loading Boost.Python made bindings is horribly slow? In-Reply-To: <406DA90F.23013.18865357@localhost> References: <406DA90F.23013.18865357@localhost> Message-ID: <406DB9AA.7020807@unity.ncsu.edu> Niall Douglas wrote: >With GCC v3.4 from CVS, I finally have build time down to only nine >hours now rather than the 20+ it was with GCC v3.3. However, when >loading the bindings into python on Linux v2.4 kernel it takes well >over six minutes. > >Is anyone else having this problem? I don't think it's BPL's fault >really as on Windows it only takes a second or two so I'm guessing >it's the dynamic linker. I'd imagine that that prebinding linkage >feature of newer Linuces will fix this problem - in the meantime, any >suggestions? > > > You most certainly can minimize the exports. The best way to do this is to segregate all of the classes and functions that you want to share between modules (python or otherwise) into a separate .so, to be placed in one of the standard library locations. Then, place all of your Python export code ( creating class_<>, def, and so on) in the module to be loaded from Python. Next, write a symbol map file. The complete specification for these things is fairly complex, but since you have segregated out all of your shared stuff, what you need is very simple (see the GNU ld manual for more). They are vaguely similar (in functionality) to .def files in Windows. My module defines BOOST_PYTHON_MODULE(cvisual) and the resulting map file looks like this: (file named linux-symbols.map) { global: initcvisual; local: *; }; Now, the only function that is exported is extern "C" initcvisual() and the resulting cvisual.so file size is very significantly reduced. (from about 3MB to 1.5 MB for me). You have to add the option -Wl,--version-script=$(MAPFILE_NAME) to the link command when invoking the linker via G++ to make use of this file. HTH, Jonathan Brandmeyer From s_sourceforge at nedprod.com Fri Apr 2 21:07:57 2004 From: s_sourceforge at nedprod.com (Niall Douglas) Date: Fri, 02 Apr 2004 20:07:57 +0100 Subject: [C++-sig] Loading Boost.Python made bindings is horribly slow? In-Reply-To: <0a25731b1342271ceab98f36271d7dd8406db042@Orthosoft.ca> Message-ID: <406DC81D.19888.18FFA330@localhost> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2 Apr 2004 at 13:10, Stefan Seefeld wrote: > > (Here's a thought - GCC outputs far too many symbols into the public > > ELF table - if GCC had a dllexport/dllimport mechanism like on > > Windows, the amount of symbols needing to be linked could be reduced > > very significantly indeed. As it so happens, someone submitted a > > The simplest thing to do (in this particular context) is to put > everything into an anonymous namespace. Everything but the actual > python module init function ('BOOST_PYTHON_MODULE()'). That should > restrict the number of the exposed symbols to a strict minimum. No, most of the symbols come from instantiations of items from the BPL headers and these exist inside the boost::python namespace. The number of symbols generated by the class wrappers (the only thing you can put inside an anonymous namespace as the Export_XXX function must remain linkable) is no more than 5% at most I'd guess. If you take a dump using elfdump (I think), you'll see what I mean. Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQG26DsEcvDLFGKbPEQJHhACfXmMPexBHp6ZoFWNCti7jlYwAuTUAoPsl r0uwQen+29jbn10GKl+dltc7 =zmWC -----END PGP SIGNATURE----- From dave at boost-consulting.com Fri Apr 2 21:23:27 2004 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 02 Apr 2004 14:23:27 -0500 Subject: [C++-sig] Re: Accessing instances of C++ objects in Python - Plus more References: <3.0.6.32.20040402093424.009e8458@mail.digipen.edu> Message-ID: Mike Gonzales writes: > Thank you for your input - I am incorporating it to make my code neater and > more precise. Anyway, I guess my question devolves to this: How do I > access a C++ variable from python, and what steps do I need to take? You can either expose it as a static member of a class using def_readonly or def_readwrite: class_("X") .def_readwrite("foo", &some_variable) ; and access it as: >>> your_extension_module.X.foo or you can enter it in your module dictionary: scope().attr("foo") = object(boost::ref(some_variable)); and access it as: >>> your_extension_module.foo > The most obvious hack that I thought of was to try and manually > insert the python-wrapped C++ variable into the main module's > namespace dictionary; Is there a better way than that? HTH, -- Dave Abrahams Boost Consulting www.boost-consulting.com From ericries at speakeasy.net Fri Apr 2 22:44:25 2004 From: ericries at speakeasy.net (Eric Ries) Date: Fri, 02 Apr 2004 20:44:25 +0000 Subject: [C++-sig] wchar_t bindings for Boost.Python Message-ID: David Abrahams suggested that I post my message below here, which I mistakenly sent directly to him originally. I am happy to provide source code if there is interest; let me know. Thanks! Eric -----Original Message----- From: David Abrahams [mailto:dave at boost-consulting.com] Sent: Friday, April 2, 2004 08:31 PM To: 'Eric Ries' Subject: Re: [C++-sig] Re: Boost.Python problems "Eric Ries" writes: > Hello, > > I am a brand new Boost.Python user, and I am very impressed. So first > off, thanks for the awesome work that has gone into Boost.Python and > Boost generally. Glad you like it! > I have been using Pyste to generate Boost.Python bindings for C++, and > noticed that there is not automatic support for wchar_t -> py unicode > conversions for C++ functions that return that type. > > While searching for a solution, I came across an email of yours > (http://mail.python.org/pipermail/c++-sig/2003-March/003633.html) in > which you wrote: > >>> 4. There are no converters for std::wstring or wchar_t. >> >>Good point. Care to contribute some? > > So I figured I would give it a try. I was pleasantly surprised by how > easy it was. That probably means that either 1) you guys have already > done this on your development branch (I am on a vanilla 1.31.0 intall) > or 2) my solution was easy because it does not generalize to other > compilers. > > But, on the off chance that it might be useful to you, I wanted to > write and see if you would like me to send in a patch. I modified > Pyste as well as several boost.python hpp files. I have only tested on > MSVC6. First, please in the future always post things like this on the C++-sig. There are other people there who ought to see your post too. We already have wstring converters, but you could contribute converters for wchar_t and I'd be happy to have them. I don't know why you'd have to change Pyste, but that's not my department (another reason to post on the C++-sig). Cheers, -- Dave Abrahams Boost Consulting www.boost-consulting.com From s_sourceforge at nedprod.com Sat Apr 3 00:22:35 2004 From: s_sourceforge at nedprod.com (Niall Douglas) Date: Fri, 02 Apr 2004 23:22:35 +0100 Subject: [C++-sig] Loading Boost.Python made bindings is horribly slow? In-Reply-To: <406DB9AA.7020807@unity.ncsu.edu> References: <406DA90F.23013.18865357@localhost> Message-ID: <406DF5BB.31402.19B1D114@localhost> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2 Apr 2004 at 14:06, Jonathan Brandmeyer wrote: > { > global: > initcvisual; > local: > *; > }; > > Now, the only function that is exported is extern "C" initcvisual() > and the resulting cvisual.so file size is very significantly reduced. > (from about 3MB to 1.5 MB for me). You have to add the option > -Wl,--version-script=$(MAPFILE_NAME) to the link command when invoking > the linker via G++ to make use of this file. Useful trick this - thanks for pointing it out. It took me some time (GNU ld seems very finickity about the grammar being exactly broken in the right places) but I got this: TNFOXP1 { global: extern "C++" { initTnFOX; boost::* }; local: extern "C++" { * }; }; As I patch Boost.Python in various places to enable multithreading support, it makes sense to bundle my patched BPL with the bindings. This is why I needed to work out how to specify C++ identifiers. This method is not perfect on anything more varied eg; a general purpose library. Studying the dynamic symbol table using "nm -C -D " shows symbols which match the wildcards leak out in a chance-like fashion. What we really need is Windows-style specifiers embedded directly into the source. If I get some time this week, I may give a bash at patching GCC. Then we can simply reuse the macros defining Windows DLL support to do the same on POSIX. Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQG3nrMEcvDLFGKbPEQIClQCgrt1p2c6Z/WoviTyMbW0UcSPXDScAnj59 Ltfgs0FjUey+eNYScSC9HJcQ =ZK8C -----END PGP SIGNATURE----- From grafik.list at redshift-software.com Sat Apr 3 01:17:59 2004 From: grafik.list at redshift-software.com (Rene Rivera) Date: Fri, 02 Apr 2004 17:17:59 -0600 Subject: [C++-sig] Loading Boost.Python made bindings is horribly slow? In-Reply-To: <406DF5BB.31402.19B1D114@localhost> References: <406DA90F.23013.18865357@localhost> <406DF5BB.31402.19B1D114@localhost> Message-ID: <406DF4A7.1030900@redshift-software.com> Niall Douglas wrote: > On 2 Apr 2004 at 14:06, Jonathan Brandmeyer wrote: >> >>Now, the only function that is exported is extern "C" initcvisual() >>and the resulting cvisual.so file size is very significantly reduced. >>(from about 3MB to 1.5 MB for me). You have to add the option >>-Wl,--version-script=$(MAPFILE_NAME) to the link command when invoking >>the linker via G++ to make use of this file. > > > Useful trick this - thanks for pointing it out. > If I get some time this week, I may give a bash at patching GCC. Then > we can simply reuse the macros defining Windows DLL support to do the > same on POSIX. I explain how I do the equivalent of dllexport on Linux+GCC on this message... http://sourceforge.net/mailarchive/forum.php?thread_id=2793609&forum_id=34093 Doesn't require changing GCC, but is ELF only, and requires recent enough binutils. One adds something like such.., to each method to export: __attribute__ ((section (".text.dll"))) -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com - 102708583/icq From dave at boost-consulting.com Sat Apr 3 01:03:18 2004 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 02 Apr 2004 18:03:18 -0500 Subject: [C++-sig] Re: Virtual member functions cause exception in get_derived_class_object References: <001101c41734$5a143af0$64b3fea9@Presario2500> Message-ID: "David Rostowsky" writes: > Attached are the files in question. Thanks for the direction. Im a newbie to > these lists. In my original post I was trying to post the minimal that > failed, so hence I wanted to give everything in the follow-up. Ill try to > find a happier middle ground for future posts. Can you try to answer the other questions I asked in my reply? -- Dave Abrahams Boost Consulting www.boost-consulting.com From s_sourceforge at nedprod.com Sat Apr 3 03:44:43 2004 From: s_sourceforge at nedprod.com (Niall Douglas) Date: Sat, 03 Apr 2004 02:44:43 +0100 Subject: [C++-sig] Loading Boost.Python made bindings is horribly slow? In-Reply-To: <406DF4A7.1030900@redshift-software.com> References: <406DF5BB.31402.19B1D114@localhost> Message-ID: <406E251B.11466.1A6AE0C0@localhost> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2 Apr 2004 at 17:17, Rene Rivera wrote: > > If I get some time this week, I may give a bash at patching GCC. > > Then we can simply reuse the macros defining Windows DLL support to > > do the same on POSIX. > > I explain how I do the equivalent of dllexport on Linux+GCC on this > message... > > http://sourceforge.net/mailarchive/forum.php?thread_id'93609&forum_i > d4093 > > Doesn't require changing GCC, but is ELF only, and requires recent > enough binutils. One adds something like such.., to each method to > export: Clever hack to solve the problem, but I think it inexcusable that GCC doesn't do this properly already especially as ELF has the functionality and has had it for ages. OTOH you shouldn't moan if you're in a position to fix it yourself, so I won't - I'll just go do it. If the patch were accepted to mainline CVS, you'd simply replace all __declspec(dllexport) with __attribute__ ((visibility("default"))) and compile with the new -fvisibility=hidden command line arg. Hey presto, link times drop substantially, executables become much smaller and faster and no more symbol clashes. ie; life gets much better on POSIX like platforms and we all celebrate! :) Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQG4XC8EcvDLFGKbPEQIr6ACgvqhqs8DB5cl5gLp5+omfO4flUiwAn0MA L3+Ay5n+xxnT6jH+WL+EjT9T =jQL6 -----END PGP SIGNATURE----- From dave at boost-consulting.com Sat Apr 3 16:35:47 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sat, 03 Apr 2004 09:35:47 -0500 Subject: [C++-sig] Re: Loading Boost.Python made bindings is horribly slow? References: <406DF5BB.31402.19B1D114@localhost> <406E251B.11466.1A6AE0C0@localhost> Message-ID: "Niall Douglas" writes: > On 2 Apr 2004 at 17:17, Rene Rivera wrote: > >> > If I get some time this week, I may give a bash at patching GCC. >> > Then we can simply reuse the macros defining Windows DLL support to >> > do the same on POSIX. >> >> I explain how I do the equivalent of dllexport on Linux+GCC on this >> message... >> >> http://sourceforge.net/mailarchive/forum.php?thread_id'93609&forum_i >> d4093 >> >> Doesn't require changing GCC, but is ELF only, and requires recent >> enough binutils. One adds something like such.., to each method to >> export: > > Clever hack to solve the problem, but I think it inexcusable that GCC > doesn't do this properly already especially as ELF has the > functionality and has had it for ages. OTOH you shouldn't moan if > you're in a position to fix it yourself, so I won't - I'll just go do > it. > > If the patch were accepted to mainline CVS, you'd simply replace all > __declspec(dllexport) with __attribute__ ((visibility("default"))) > and compile with the new -fvisibility=hidden command line arg. Hey > presto, link times drop substantially, executables become much > smaller and faster and no more symbol clashes. > > ie; life gets much better on POSIX like platforms and we all > celebrate! :) I'm looking forward to that! Something to watch out for if you do this: if __attribute__ ((visibility("default"))) applies to classes, you'll have to decide how to handle member functions that are declared inline. If you don't handle it the same way as Win32 compilers do, you could introduce subtle client portability problems. Of course, if it doesn't apply to classes, your feature won't be nearly so useful ;-) -- Dave Abrahams Boost Consulting www.boost-consulting.com From seefeld at sympatico.ca Sat Apr 3 17:47:44 2004 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Sat, 03 Apr 2004 10:47:44 -0500 Subject: [C++-sig] Re: Loading Boost.Python made bindings is horribly slow? In-Reply-To: References: <406DF5BB.31402.19B1D114@localhost> <406E251B.11466.1A6AE0C0@localhost> Message-ID: <406EDCA0.6000603@sympatico.ca> Sorry for the offtopic question, but this thread reminds me of the old question: Has there been any attempt to standardize (read: put into the C and C++ languages) tools to express desired visibility similar to the proprietary dllexport / __attribute__ mechanisms ? Thanks, Stefan From s_sourceforge at nedprod.com Sat Apr 3 20:21:43 2004 From: s_sourceforge at nedprod.com (Niall Douglas) Date: Sat, 03 Apr 2004 19:21:43 +0100 Subject: Exception handling as thrown from BPL so (was: Re: [C++-sig] Re: Loading Boost.Python made bindings is horribly slow?) In-Reply-To: Message-ID: <406F0EC7.8064.1DFBA891@localhost> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Related to my previous post, I am now finding that if I throw an exception from within my python extension module, the client program won't catch it. This has started happening since I implemented that symbol versioning trick previously suggested. I'm guessing that the type info symbols are getting stripped and so it's unable to match the exception type with the catch type. Am I right in this? It's very hard to prove as it's a 130Mb shared object (this is *without* debug info) and it's horribly mangles my computer when running in gdb. If I am right (please confirm), does anyone know how RTTI symbols are mangled precisely so I can keep them from getting stripped? (Roll on declspec() for GCC!) Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQG8AucEcvDLFGKbPEQI83gCg9wdlBKZW6O4lcBOKU8TMHjKNEV0AoO7R DTLo/iWNrZ5uOolZ2czIvc7E =oPYE -----END PGP SIGNATURE----- From s_sourceforge at nedprod.com Sat Apr 3 20:30:04 2004 From: s_sourceforge at nedprod.com (Niall Douglas) Date: Sat, 03 Apr 2004 19:30:04 +0100 Subject: [C++-sig] Re: Loading Boost.Python made bindings is horribly slow? In-Reply-To: Message-ID: <406F10BC.19337.1E034EE1@localhost> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 3 Apr 2004 at 9:35, David Abrahams wrote: > I'm looking forward to that! Something to watch out for if you do > this: if > > __attribute__ ((visibility("default"))) > > applies to classes, you'll have to decide how to handle member > functions that are declared inline. If you don't handle it the same > way as Win32 compilers do, you could introduce subtle client > portability problems. Of course, if it doesn't apply to classes, your > feature won't be nearly so useful ;-) No, it will apply to classes. The patch at http://gcc.gnu.org/bugzilla/show_bug.cgi?id?83 by someone expert at modifying the GCC internals points me the right way. It'll also use identical syntax to MSVC, despite the misgivings of the GCC crew. I was going to always mark inline functions as hidden. Also all templates. This creates code bloat but gives far faster loading & linking times plus less coupling problems between modules. It also follows MSVC. As my previous post pointed out, RTTI is going to be a headache - if you remember Dave some months ago you agreed that some form of being able to filter out the RTTI would be useful in reducing binary size - I figure that if most of the RTTI is not public, it lets the compiler + linker know it can be elided (whether the GNU tools can already do this is unknown). Debugging this 130Mb shared object is a royal PITA. I'm getting tempted to delay release of v0.75 until GCC is patched, but then no one else could make my library without my patch :( Cheers, Niall -----BEGIN PGP SIGNATURE----- Version: idw's PGP-Frontend 4.9.6.1 / 9-2003 + PGP 8.0.2 iQA/AwUBQG8CrMEcvDLFGKbPEQInWACeNBlO+ea6PQefSkC8Vi/AKmQLVV8An0pQ CP7e8gXgtrYj1WpjoXKf8ZIg =JqJK -----END PGP SIGNATURE----- From dave at boost-consulting.com Sat Apr 3 20:54:59 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sat, 03 Apr 2004 13:54:59 -0500 Subject: [C++-sig] Re: Exception handling as thrown from BPL so References: <406F0EC7.8064.1DFBA891@localhost> Message-ID: "Niall Douglas" writes: > Related to my previous post, I am now finding that if I throw an > exception from within my python extension module, the client program > won't catch it. This has started happening since I implemented that > symbol versioning trick previously suggested. > > I'm guessing that the type info symbols are getting stripped and so > it's unable to match the exception type with the catch type. Am I > right in this? Very probably. > It's very hard to prove as it's a 130Mb shared object > (this is *without* debug info) and it's horribly mangles my computer > when running in gdb. Make a reduced test case then. -- Dave Abrahams Boost Consulting www.boost-consulting.com From dave at boost-consulting.com Sun Apr 4 04:20:42 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sat, 03 Apr 2004 21:20:42 -0500 Subject: [C++-sig] Re: conversion of python dictionary to shared_ptrs References: <93618711046370@webmail.iastate.edu> <200404020301.i3231v3N5674255@vracs001.vrac.iastate.edu> Message-ID: "aashish" writes: > Hi, > > I need to pass a dictionary from python to C++ function which is defined as > > void setVariables(ValueMapPtr variables); > > where I have typedef ValueMapPtr as > > //a shared ptr of a value map > typedef boost::shared_ptr< ValueMap > ValueMapPtr; > > // And > typedef std::map< std::string, std::vector > ValueMap; > > > Is it possible?? Yes. > Do I need to use register_ptr_to_python ? No; if it's truly a Python dictionary on the Python side, you need to build a custom rvalue from-python converter, per http://www.boost.org/libs/python/doc/v2/faq.html#question2 http://article.gmane.org/gmane.comp.python.c%2B%2B/2161 I wish it were easier; sorry! -- Dave Abrahams Boost Consulting www.boost-consulting.com From RaoulGough at yahoo.co.uk Sun Apr 4 12:31:37 2004 From: RaoulGough at yahoo.co.uk (Raoul Gough) Date: Sun, 04 Apr 2004 11:31:37 +0100 Subject: [C++-sig] Re: Newbie needs advice for decision. References: <4069558C.4060703@freenet.de> Message-ID: Billy Gnosis writes: > Hi I want to extend and more important embed Python in my Programs. > > 1. What is the best (easiest way) to extend Python with C++ ? Swig or > Boost.Python, or is there something easier ? I haven't used Swig, so I can't really compare them. Using Boost.Python is pretty easy once you get into it, but it will probably take a little bit of effort to get there (see the tutorial to get an idea). > > 2. What is the best (easiest way) to embed Python in a C++ Program ? > Boost.Python, the Python/C Api, or is there something easier out there > ? Boost.Python doesn't provide a lot of support for embedding Python. It does basic stuff like managing reference counts for you and so on, but doesn't currently wrap much of the Python/C API. > > > 3. General Question: I played a little bit wit the high level > embedding of the python interpreter in my C++ Program (#include > "Python.h", Py_Initialize()...etc.etc...). Can I somehow compile and > link everything statically ? Meaning, I have one binary in the end, > that already includes all necessary parts of the python interpreter - > when I give this file somebody else, this should be enough to run it > no installation of the python interpreter necessary, etc... > > Is this possible ? I guess anything is possible since you have access to the Python source code and could relink it anyway you want, given enough effort. You'd probably do better to ask about this on the main Python discussion group, since it's not specifically C++ related. I'm not sure about any legal/licensing issues of doing this, and how much of the Python library (i.e. .py or .pyc files) the interpreter needs to have to get going. -- Raoul Gough. export LESS='-X' From RaoulGough at yahoo.co.uk Sun Apr 4 12:56:15 2004 From: RaoulGough at yahoo.co.uk (Raoul Gough) Date: Sun, 04 Apr 2004 11:56:15 +0100 Subject: [C++-sig] Re: conversion of python dictionary to shared_ptrs References: <93618711046370@webmail.iastate.edu> <200404020301.i3231v3N5674255@vracs001.vrac.iastate.edu> Message-ID: David Abrahams writes: > "aashish" writes: > >> Hi, >> >> I need to pass a dictionary from python to C++ function which is defined as >> >> void setVariables(ValueMapPtr variables); >> >> where I have typedef ValueMapPtr as >> >> //a shared ptr of a value map >> typedef boost::shared_ptr< ValueMap > ValueMapPtr; >> >> // And >> typedef std::map< std::string, std::vector > ValueMap; >> >> >> Is it possible?? > > Yes. > >> Do I need to use register_ptr_to_python ? > > No; if it's truly a Python dictionary on the Python side, you need to > build a custom rvalue from-python converter, per > > http://www.boost.org/libs/python/doc/v2/faq.html#question2 > http://article.gmane.org/gmane.comp.python.c%2B%2B/2161 > > I wish it were easier; sorry! There is an interesting trade off here, which I've thought about on a few occasions. If you want to share a container between Python and C++ code, I believe you have three choices: 1. Use a Python container and access it from C++ using the Python/C APIs like (e.g.) PyDict_GetItem 2. Use a C++ container and access it from Python using (e.g.) the indexing suite. 3. Use native containers on each side and convert from/to the different types at the interface. In terms of performance, which one is appropriate depends on the application, where most of the processing takes place and probably also on the size of the container. Option 1. means dealing with PyObjects from C++, instead of having the compiler know the static types of the elements. It introduces overheads on the C++ side, so isn't much good if the heavy processing is done here. Option 2. means the C++ code knows the static types of the elements, which gives you the fastest possible code for what you do in C++. It introduces overheads on the Python side, since every operation on the container requires a call through the Boost.Python code. For example, this wouldn't make much sense if you spend all the time filling the container from Python and then do one simple operation on it in C++. Option 3. means converting the whole container for each call (either from a C++ container to Python or vice versa). I guess it also introduces a problem if you want code on the non-native side to modify the container, since you'd then have to convert the container back and replace the original. As for which is the easiest to use, I would say probably option 2 would be, except that I haven't yet documented the new indexing suite properly, and I don't think the original suite has much std::map support. -- Raoul Gough. export LESS='-X' From dave at boost-consulting.com Sun Apr 4 17:44:58 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sun, 04 Apr 2004 11:44:58 -0400 Subject: [C++-sig] Re: conversion of python dictionary to shared_ptrs References: <93618711046370@webmail.iastate.edu> <200404020301.i3231v3N5674255@vracs001.vrac.iastate.edu> Message-ID: Raoul Gough writes: > David Abrahams writes: > >> "aashish" writes: >> >>> Hi, >>> >>> I need to pass a dictionary from python to C++ function which is defined as >>> >>> void setVariables(ValueMapPtr variables); >>> >>> where I have typedef ValueMapPtr as >>> >>> //a shared ptr of a value map >>> typedef boost::shared_ptr< ValueMap > ValueMapPtr; >>> >>> // And >>> typedef std::map< std::string, std::vector > ValueMap; >>> >>> >>> Is it possible?? >> >> Yes. >> >>> Do I need to use register_ptr_to_python ? >> >> No; if it's truly a Python dictionary on the Python side, you need to >> build a custom rvalue from-python converter, per >> >> http://www.boost.org/libs/python/doc/v2/faq.html#question2 >> http://article.gmane.org/gmane.comp.python.c%2B%2B/2161 >> >> I wish it were easier; sorry! > > There is an interesting trade off here, which I've thought about on a > few occasions. If you want to share a container between Python and C++ > code, I believe you have three choices: > > 1. Use a Python container and access it from C++ using the Python/C > APIs like (e.g.) PyDict_GetItem > > 2. Use a C++ container and access it from Python using (e.g.) the > indexing suite. > > 3. Use native containers on each side and convert from/to the > different types at the interface. > > In terms of performance, which one is appropriate depends on the > application, where most of the processing takes place and probably > also on the size of the container. > > Option 1. means dealing with PyObjects from C++, instead of having the > compiler know the static types of the elements. It introduces > overheads on the C++ side, so isn't much good if the heavy processing > is done here. > > Option 2. means the C++ code knows the static types of the elements, > which gives you the fastest possible code for what you do in C++. It > introduces overheads on the Python side, since every operation on the > container requires a call through the Boost.Python code. I'm not convinced these overheads you're referring to are real. Accessing a native Python dict or list from Python requires a call through Python's 'C' code that's much like Boost.Python's C++ code. There's probably one extra level of function indirection, but I doubt that counts for much in most cases. > For example, this wouldn't make much sense if you spend all the time > filling the container from Python and then do one simple operation > on it in C++. Probably the reason, though, is the amount of effort you expend writing wrapping code in that case. > Option 3. means converting the whole container for each call (either > from a C++ container to Python or vice versa). I guess it also > introduces a problem if you want code on the non-native side to modify > the container, since you'd then have to convert the container back and > replace the original. Right. > As for which is the easiest to use, I would say probably option 2 > would be, except that I haven't yet documented the new indexing suite > properly, and I don't think the original suite has much std::map > support. Mmm, I'm eagerly awaiting those docs. -- Dave Abrahams Boost Consulting www.boost-consulting.com From RaoulGough at yahoo.co.uk Mon Apr 5 00:36:28 2004 From: RaoulGough at yahoo.co.uk (Raoul Gough) Date: Sun, 04 Apr 2004 23:36:28 +0100 Subject: [C++-sig] Re: conversion of python dictionary to shared_ptrs References: <93618711046370@webmail.iastate.edu> <200404020301.i3231v3N5674255@vracs001.vrac.iastate.edu>

Message-ID: David Abrahams writes: > Raoul Gough writes: [snip] >> Option 2. means the C++ code knows the static types of the elements, >> which gives you the fastest possible code for what you do in C++. It >> introduces overheads on the Python side, since every operation on the >> container requires a call through the Boost.Python code. > > I'm not convinced these overheads you're referring to are real. > Accessing a native Python dict or list from Python requires a call > through Python's 'C' code that's much like Boost.Python's C++ code. > There's probably one extra level of function indirection, but I doubt > that counts for much in most cases. Hmmm... I'd always assumed there was more to it than just an extra indirection, doesn't it have to search for a matching overload and do type compatibility checking as well? In particular, the purely Python solution doesn't care what kind of object you insert in a container, so it doesn't have to check types or look up a from-Python converter. I must admit I never really looked into in much detail, but I did see a substantial performance difference between filling a Python list from Python and filling a std::vector from Python (about a factor of five). Is this more than you would expect? On the other hand, C++ code searching the vector or sorting it is about fifteen times faster than Python code doing the same on a Python list, so it really does depend on the mix of operations you're doing. > >> For example, this wouldn't make much sense if you spend all the time >> filling the container from Python and then do one simple operation >> on it in C++. > > Probably the reason, though, is the amount of effort you expend > writing wrapping code in that case. > >> Option 3. means converting the whole container for each call (either >> from a C++ container to Python or vice versa). I guess it also >> introduces a problem if you want code on the non-native side to modify >> the container, since you'd then have to convert the container back and >> replace the original. > > Right. > >> As for which is the easiest to use, I would say probably option 2 >> would be, except that I haven't yet documented the new indexing suite >> properly, and I don't think the original suite has much std::map >> support. > > Mmm, I'm eagerly awaiting those docs. I've recently managed to get some paid employment, so I've had to put this work on the back burner for a while. I'm still hoping to get the docco ready by the end of next week (April 16) in time for the Python conference in Oxford. -- Raoul Gough. export LESS='-X' From dave at boost-consulting.com Mon Apr 5 03:36:45 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sun, 04 Apr 2004 21:36:45 -0400 Subject: [C++-sig] Re: conversion of python dictionary to shared_ptrs References: <93618711046370@webmail.iastate.edu> <200404020301.i3231v3N5674255@vracs001.vrac.iastate.edu>

Message-ID: <4078C76D.2050507@fastmail.fm> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Faheem Mitha wrote: | Daniel Holth just wrote that distutils works for him on Linux | (apparently he uses Gentoo). He mentioned a Python extension | package of his called shoutpy which uses boost.python and | distutils, and it builds on Debian with no problem. Therefore, I | might give this a try. I'm very glad you've enjoyed my "exemplary" module. And I've also figured out boost.build v1. Definitely not harder to figure out than automake or even make. It's a very edifying way to build my little C++ example, too. Version 0.5.2 now includes a Jamfile that builds the module. http://dingoskidneys.com/shoutpy/ shoutpy is a wrapper that lets you use Python to send compressed audio to an icecast server for internet radio broadcast. The wrapper uses three files: a header and C++ file to wrap libshout2 in a C++ class, and the boost.python declarations in the third. http://www.xiph.org/~brendan/shout-python/ wraps the same library using the Python C api directly; my boost.python binary is 29 times larger than that when compiled by distutils and the boost-jam debug build is 165 times larger. When you run 'strip' on shoutpy.so the extension is only 18 times larger than the C version of almost the same thing. But as we know the boost.python wrapper gets written much more quickly and in a less dodgy way than often ad-hoc direct-to-Python-API code, and if we didn't care about programmer efficiency we wouldn't be using Python in the first place. - - Daniel Holth -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFAeMdsVh4W2pVfoMsRAtC+AJ0WRzoowFRNKUX6TYr6NtdB2qG/kwCgzWIf pUO8JgWIZVh6nn4qyO179AU= =ggN1 -----END PGP SIGNATURE----- From dave at boost-consulting.com Sun Apr 11 13:56:43 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sun, 11 Apr 2004 07:56:43 -0400 Subject: [C++-sig] Re: help building hello world example on Debian References:

<4078C76D.2050507@fastmail.fm> Message-ID: Daniel Holth writes: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Faheem Mitha wrote: > > | Daniel Holth just wrote that distutils works for him on Linux > | (apparently he uses Gentoo). He mentioned a Python extension > | package of his called shoutpy which uses boost.python and > | distutils, and it builds on Debian with no problem. Therefore, I > | might give this a try. > > I'm very glad you've enjoyed my "exemplary" module. And I've also > figured out boost.build v1. Definitely not harder to figure out than > automake or even make. It's a very edifying way to build my little C++ > example, too. I just updated the CVS with a better-commented example directory; see http://boost-consulting.com/boost/libs/python/example/README, which should be updated within the hour. Daniel, it'd be a great idea for you to make sure that your shoutpy example uses the same techniques, if you're going to be passing it around as an example. Why did you do this? You don't need it: # Include definitions needed for Python modules SEARCH on python.jam = $(BOOST_BUILD_PATH) ; > Version 0.5.2 now includes a Jamfile that builds the module. > http://dingoskidneys.com/shoutpy/ > > shoutpy is a wrapper that lets you use Python to send compressed audio > to an icecast server for internet radio broadcast. The wrapper uses > three files: a header and C++ file to wrap libshout2 in a C++ class, > and the boost.python declarations in the third. Nice! What can one do with this library, in lay terms? > http://www.xiph.org/~brendan/shout-python/ wraps the same library > using the Python C api directly; my boost.python binary is 29 times > larger than that when compiled by distutils and the boost-jam debug > build is 165 times larger. When you run 'strip' on shoutpy.so the > extension is only 18 times larger than the C version of almost the > same thing. But as we know the boost.python wrapper gets written much > more quickly and in a less dodgy way than often ad-hoc > direct-to-Python-API code, and if we didn't care about programmer > efficiency we wouldn't be using Python in the first place. Well, yeah, and did you try a release build? -- Dave Abrahams Boost Consulting www.boost-consulting.com From faheem at email.unc.edu Sun Apr 11 17:20:28 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sun, 11 Apr 2004 15:20:28 +0000 (UTC) Subject: [C++-sig] Re: help building hello world example on Debian References:

<4078C76D.2050507@fastmail.fm> Message-ID: <40797210.7030903@fastmail.fm> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Abrahams wrote: | I just updated the CVS with a better-commented example directory; | see http://boost-consulting.com/boost/libs/python/example/README, | which should be updated within the hour. Daniel, it'd be a great | idea for you to make sure that your shoutpy example uses the same | techniques, if you're going to be passing it around as an example. Good, I'll check mine. Thanks for the improved example. It is better, I've just incorporated your Jamrules into shoutpy. shoutpy is an instance of a simple external Boost.Python extension, optionally using boost-build or distutils-, and it depends on a non-boost library. I would like to hear about more libraries that meet this criteria. warning: shoutpy's boost-jam setup may not work right now for you. Must I distribute the boost license along with my program because I've used your example? | Why did you do this? You don't need it: | | # Include definitions needed for Python modules SEARCH on | python.jam = $(BOOST_BUILD_PATH) ; It's there because it was in project.zip , no other reason. |> http://dingoskidneys.com/shoutpy/ |> |> shoutpy is a wrapper that lets you use Python to send compressed |> audio | Nice! What can one do with this library, in lay terms? You can use shoutpy to send Ogg/Vorbis or mp3 music to an Internet radio station. | Well, yeah, and did you try a release build? Just did... the module's binary is 135 kilobytes. That's about half the size of the distutils build, very good! What are the advantages to using this: ~