Message-ID: <87zlog5wu2.fsf@mcbain.luannocracy.com> on Thu Jul 17 2008, Gennadiy Rozental wrote: > Gennadiy Rozental gmail.com> writes: > > Forgot to mention two critical bits of info > >> class PythonBase : public Base { >> public: > // release python object > ~PythonBase() { bp::decref( m_object ) } > ... That looks like a recipe for double-deletion. When the bp::object m_object goes out of scope, it will take care of that part. >> Base* >> PythonFactory::create(...) >> { >> bp::object obj = m_py_type(); >> PythonBase* ptr = bp::extract( obj ); >> >> ptr->init( obj ); > > // make sure python object is not destroyed once we leave this scope > bp::incref( obj ); Oh, well in that case, it adds nothing. You may as well leave the incref/decref out. >> return ptr; >> } > > Gennadiy > > _______________________________________________ > C++-sig mailing list > C++-sig at python.org > http://mail.python.org/mailman/listinfo/c++-sig > -- Dave Abrahams BoostPro Computing http://www.boostpro.com From dave at boostpro.com Thu Jul 17 17:56:52 2008 From: dave at boostpro.com (David Abrahams) Date: Thu, 17 Jul 2008 11:56:52 -0400 Subject: [C++-sig] differencein memory management In-Reply-To: (Gennadiy Rozental's message of "Thu, 17 Jul 2008 07:06:35 +0000 (UTC)") References: <87tzepzbcy.fsf@mcbain.luannocracy.com> Message-ID: <87vdz45wsb.fsf@mcbain.luannocracy.com> on Thu Jul 17 2008, Gennadiy Rozental wrote: > David Abrahams boostpro.com> writes: > >> > class_,... >> > class_,... >> > class_> >> The first two are identical. The third one leaks a C++ A when the >> corresponding Python A loses its last reference. > > I guess I need the later - I want to manage C++ object lifecycle myself. But in > this case I can't get from A* back to the python object, right? Not unless you add some machinery to do so explicitly. -- Dave Abrahams BoostPro Computing http://www.boostpro.com From rogeeff at gmail.com Thu Jul 17 18:19:05 2008 From: rogeeff at gmail.com (Gennadiy Rozental) Date: Thu, 17 Jul 2008 16:19:05 +0000 (UTC) Subject: [C++-sig] set python attribute from C++ References: <87skuahg80.fsf@mcbain.luannocracy.com> <87k5fl1smh.fsf@mcbain.luannocracy.com>

<87zlog5wu2.fsf@mcbain.luannocracy.com> Message-ID: David Abrahams boostpro.com> writes: > >> Base* > >> PythonFactory::create(...) > >> { > >> bp::object obj = m_py_type(); > >> PythonBase* ptr = bp::extract( obj ); > >> > >> ptr->init( obj ); > > > > // make sure python object is not destroyed once we leave this scope > > bp::incref( obj ); > > Oh, well in that case, it adds nothing. You may as well leave the > incref/decref out. No. I can't (or at least it does not work for me). Without it python object is destroyed immediately once we leave create function scope and ptr become dangling pointer. Gennadiy From Mark.English at rbccm.com Thu Jul 17 18:04:57 2008 From: Mark.English at rbccm.com (English, Mark) Date: Thu, 17 Jul 2008 17:04:57 +0100 Subject: [C++-sig] set python attribute from C++ Message-ID: <75EF75D50F2523408605799EE2DD030E1DA0DD@SXLP-002.fg.rbc.com> > -----Original Message----- > From: c++-sig-bounces at python.org > [mailto:c++-sig-bounces at python.org] On Behalf Of Gennadiy Rozental > Sent: 17 July 2008 08:04 > To: c++-sig at python.org > Subject: Re: [C++-sig] set python attribute from C++ > > > On python side I've can now derive from PythonBase: > > class Derived(PythonBase): > pass > > Now i am seems to be able to create instances of Derived > class and set their field from c++. Does this solution make sence? I've had similar problems being forced to use other 3rd party shared pointers. There's a sample somewhere using smart_ptr somewhere which is a good starting point for discovering the issues. >From memory since I don't have the code handy, my last effort was storing a back_reference (using has_back_reference) to the Python object. As I think Mr Abrahams says, this is the wrong way round since the Python object is supposed to manage the C++ object's lifetime. It does however have the advantage of preserving Python object identity, leading to behaviour like that you describe where exposing the held type as my_smart_ptr , and holding some kind of reference to the PyObject, allows Python polymorhpism to work. However, the scenario I never got passed was this: Python code derives a class "PyFoo" from exposed C++ class "Foo". Python code creates an instance of "PyFoo" and passes it to C++ code which expects a my_smart_ptr. C++ code stores my_smart_ptr in a container for later retrieval. Python code deletes its reference to the "PyFoo" instance. "PyFoo" object reference count falls to 0 and object is no longer used on Python side. ...Time passes... Python code calls into C++ code to retrieve object stored in C++ collection of my_smart_ptr. Internally cached PyObject is returned who's reference count was decremented to 0 above. Nothing works. Alternatively a new PyObject wrapper for the C++ "Foo" instance originally contained within the "PyFoo" instance can be returned, but it will only appear as a "Foo", and not a "PyFoo". Following this route requires some mechanism to say from the C++ side that the PyObject needs to remain valid despite no references to it from Python. This then leads to issues of how to manage that additional lifetime information. All in all the holding a PyObject approach looked promising for a while, but I suspect it was a mistake. Before that I tried exposing boost::shared_ptr > as my held_type which I never really got off the ground because of the difficulty in exposing inheritance relationships with that level of indirection, so I wouldn't advocate that route either. I'm sorry that I haven't got any example code handy. I'll try to knock some up if that doesn't make enough sense, the next time I'm near a compiler. I think a better solution is a custom_deleter for your my_smart_ptr, assuming it supports custom deleters. Perhaps one that extends the one provided by boost::python. Also, we're veering away from the topic of how to set python attributes in C++. Good luck ! _______________________________________________________________________ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message without copying, using, or telling anyone about its contents. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. This message does not create or change any contract. Royal Bank of Canada and its subsidiaries accept no responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. RBC Capital Markets is a business name used by branches and subsidiaries of Royal Bank of Canada, including Royal Bank of Canada, London branch and Royal Bank of Canada Europe Limited. In accordance with English law requirements, details regarding Royal Bank of Canada Europe Limited are set out below: ROYAL BANK OF CANADA EUROPE LIMITED Registered in England and Wales 995939 Registered Address: 71 Queen Victoria Street, London, EC4V 4DE. Authorised and regulated by the Financial Service Authority. Member of the London Stock Exchange From rogeeff at gmail.com Thu Jul 17 18:20:29 2008 From: rogeeff at gmail.com (Gennadiy Rozental) Date: Thu, 17 Jul 2008 16:20:29 +0000 (UTC) Subject: [C++-sig] differencein memory management References: <87tzepzbcy.fsf@mcbain.luannocracy.com> <87vdz45wsb.fsf@mcbain.luannocracy.com> Message-ID: David Abrahams boostpro.com> writes: > > I guess I need the later - I want to manage C++ object lifecycle myself. > > But in > > this case I can't get from A* back to the python object, right? > > Not unless you add some machinery to do so explicitly. > This is exactly what I did by storing bp:object inside A. Gennadiy From dave at boostpro.com Thu Jul 17 18:29:02 2008 From: dave at boostpro.com (David Abrahams) Date: Thu, 17 Jul 2008 12:29:02 -0400 Subject: [C++-sig] differencein memory management In-Reply-To: (Gennadiy Rozental's message of "Thu, 17 Jul 2008 16:20:29 +0000 (UTC)") References: <87tzepzbcy.fsf@mcbain.luannocracy.com> <87vdz45wsb.fsf@mcbain.luannocracy.com> Message-ID: <877ibk5vap.fsf@mcbain.luannocracy.com> on Thu Jul 17 2008, Gennadiy Rozental wrote: > David Abrahams boostpro.com> writes: >> > I guess I need the later - I want to manage C++ object lifecycle myself. >> > But in >> > this case I can't get from A* back to the python object, right? >> >> Not unless you add some machinery to do so explicitly. >> > > This is exactly what I did by storing bp:object inside A. Yes -- Dave Abrahams BoostPro Computing http://www.boostpro.com From dave at boostpro.com Thu Jul 17 18:30:51 2008 From: dave at boostpro.com (David Abrahams) Date: Thu, 17 Jul 2008 12:30:51 -0400 Subject: [C++-sig] set python attribute from C++ In-Reply-To: (Gennadiy Rozental's message of "Thu, 17 Jul 2008 16:19:05 +0000 (UTC)") References: <87skuahg80.fsf@mcbain.luannocracy.com> <87k5fl1smh.fsf@mcbain.luannocracy.com>

<87zlog5wu2.fsf@mcbain.luannocracy.com> Message-ID: <873am85v7o.fsf@mcbain.luannocracy.com> on Thu Jul 17 2008, Gennadiy Rozental wrote: > David Abrahams boostpro.com> writes: >> >> Base* >> >> PythonFactory::create(...) >> >> { >> >> bp::object obj = m_py_type(); >> >> PythonBase* ptr = bp::extract( obj ); >> >> >> >> ptr->init( obj ); >> > >> > // make sure python object is not destroyed once we leave this scope >> > bp::incref( obj ); >> >> Oh, well in that case, it adds nothing. You may as well leave the >> incref/decref out. > > No. I can't (or at least it does not work for me). Without it python object is > destroyed immediately once we leave create function scope and ptr become > dangling pointer. Then m_object is a raw pointer and not a bp::object as you've described in your code examples. -- Dave Abrahams BoostPro Computing http://www.boostpro.com From amohr at pixar.com Thu Jul 17 18:46:06 2008 From: amohr at pixar.com (Alex Mohr) Date: Thu, 17 Jul 2008 09:46:06 -0700 Subject: [C++-sig] differencein memory management In-Reply-To: References: <87tzepzbcy.fsf@mcbain.luannocracy.com> <87vdz45wsb.fsf@mcbain.luannocracy.com> Message-ID: <487F774E.4090407@pixar.com> >>> this case I can't get from A* back to the python object, right? >> Not unless you add some machinery to do so explicitly. > > This is exactly what I did by storing bp:object inside A. But then the problem is when you return a custom_ptr

<87zlog5wu2.fsf@mcbain.luannocracy.com> <873am85v7o.fsf@mcbain.luannocracy.com> Message-ID: David Abrahams boostpro.com> writes: > on Thu Jul 17 2008, Gennadiy Rozental wrote: > > > David Abrahams boostpro.com> writes: > >> >> Base* > >> >> PythonFactory::create(...) > >> >> { > >> >> bp::object obj = m_py_type(); > >> >> PythonBase* ptr = bp::extract( obj ); > >> >> > >> >> ptr->init( obj ); > >> > > >> > // make sure python object is not destroyed once we leave > >> > // this scope > >> > bp::incref( obj ); > >> > >> Oh, well in that case, it adds nothing. You may as well leave > >> the incref/decref out. > > > > No. I can't (or at least it does not work for me). Without it > > python object is destroyed immediately once we leave create > > function scope and ptr become dangling pointer. > > Then m_object is a raw pointer and not a bp::object as you've > described in your code examples. Hmm. You right. I do not need it anymore. I used to need it because init method above was invoked later - after we left create function. Not anymore. I removed both incref/decref and everything works fine. Thanks Gennadiy From rogeeff at gmail.com Thu Jul 17 20:20:24 2008 From: rogeeff at gmail.com (Gennadiy Rozental) Date: Thu, 17 Jul 2008 18:20:24 +0000 (UTC) Subject: [C++-sig] differencein memory management References: <87tzepzbcy.fsf@mcbain.luannocracy.com> <87vdz45wsb.fsf@mcbain.luannocracy.com> <487F774E.4090407@pixar.com> Message-ID: Alex Mohr pixar.com> writes: > > >>> this case I can't get from A* back to the python object, right? > >> Not unless you add some machinery to do so explicitly. > > > > This is exactly what I did by storing bp:object inside A. > > But then the problem is when you return a custom_ptr

Message-ID: <7465b6170807231211p6e12a34dw562717f04b0dad71@mail.gmail.com> On Wed, Jul 23, 2008 at 9:48 PM, Gennadiy Rozental wrote: > Gennadiy Rozental gmail.com> writes: > >> >> Hi, >> >> Boost.Python throws bp::error_already_set in case any error occurred. But I > > So, there is no way to tell Boost.Python to throw my exception instead? I am not sure what are you looking for. Take a look on next links: http://www.boost.org/doc/libs/1_35_0/libs/python/doc/v2/exception_translator.html http://www.language-binding.net/pyplusplus/troubleshooting_guide/exceptions/exceptions.html May be they will help -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/ From dave at boostpro.com Thu Jul 24 03:00:11 2008 From: dave at boostpro.com (David Abrahams) Date: Wed, 23 Jul 2008 21:00:11 -0400 Subject: [C++-sig] assign a pointer to global object In-Reply-To: (Gennadiy Rozental's message of "Wed, 23 Jul 2008 18:45:40 +0000 (UTC)") References: <7465b6170807231104i2723aec2j23290a821b8aa04f@mail.gmail.com> Message-ID: <87prp4hzac.fsf@mcbain.luannocracy.com> on Wed Jul 23 2008, Gennadiy Rozental wrote: > > adding boost::ref to the > > obj.attr( "aaa" ) = boost::ref( s_global ); > > did resolve the issue, though I am quite confused why it is required. Reference semantics is generally unsafe, so we make you request it explicitly. -- Dave Abrahams BoostPro Computing http://www.boostpro.com From ericdfirst at gmail.com Thu Jul 24 16:40:52 2008 From: ericdfirst at gmail.com (Eric First) Date: Thu, 24 Jul 2008 09:40:52 -0500 Subject: [C++-sig] Wrapping std::set using Boost::Python Message-ID: <349c44440807240740v41f1688ay98de6613071e4e13@mail.gmail.com> Has anybody wrapped std::set using Boost::Python? I'm looking for the most efficient way to do this. All ideas are greatly appreciated. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roman.yakovenko at gmail.com Thu Jul 24 19:16:01 2008 From: roman.yakovenko at gmail.com (Roman Yakovenko) Date: Thu, 24 Jul 2008 20:16:01 +0300 Subject: [C++-sig] Wrapping std::set using Boost::Python In-Reply-To: <349c44440807240740v41f1688ay98de6613071e4e13@mail.gmail.com> References: <349c44440807240740v41f1688ay98de6613071e4e13@mail.gmail.com> Message-ID: <7465b6170807241016l19c9e1b4iee9b8179514798c7@mail.gmail.com> 2008/7/24 Eric First : > Has anybody wrapped std::set using Boost::Python? I'm looking for the most > efficient way to do this. All ideas are greatly appreciated. Thanks. Indexing suite v2 contains such functionality. http://www.language-binding.net/pyplusplus/documentation/containers.html Also I am not sure about "most efficient". -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/ From erictexier at gmail.com Thu Jul 24 21:49:25 2008 From: erictexier at gmail.com (Eric Texier) Date: Thu, 24 Jul 2008 12:49:25 -0700 Subject: [C++-sig] using boost::python::list .... Message-ID: <37c1dd070807241249l3f6a6a43g727c3a7ff9e100fe@mail.gmail.com> Hi, I was wondering if you someone can help me on this one. I would like to return a dictionary with all the key/value of a c++ object (a camera is this case) some of these values are list ( position = [x,y,z]). I'm tempted to allocate those list dynamically but I'm concern about who is going to manage them.(delete ...reference count) To make it short: - should I use the way I do "position" (dynamic) or the way I do for "lookat"(static).? - if I allocate the list as a value of my dictionary key, is there anything else to do in term of reference count etc...? Thanks for you help. Eric namespace bp=boost::python; bp::dict getAsDict(ICam& myCam) { bp::dict retvals; // position bp::list *plist = new bp::list(); const Imath::V3f& p = myCam.position(); plist->append(p.x); plist->append(p.y); plist->append(p.z); retvals["position"] = plist; // lookat const Imath::V3f& d = myCam.lookat(); bp::list llist; llist.append(d.x); llist.append(d.y); llist.append(d.z); retvals["direction"] = llist; return retvals; } BOOST_PYTHON_MODULE(iParts) { class_("ICam") .def("getAsDict",&getAsDict,"return all the value in a dict") ; } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff.webb at nta-inc.net Thu Jul 24 22:23:10 2008 From: jeff.webb at nta-inc.net (Jeff Webb) Date: Thu, 24 Jul 2008 15:23:10 -0500 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <200807231119.36451.meine@informatik.uni-hamburg.de> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> Message-ID: <4888E4AE.3000500@nta-inc.net> The boost::python library provides excellent support for wrapping C++ objects managed by boost::shared_ptr reference counting smart pointers. I am attempting to achieve similar functionality with a little less memory overhead by embedding the reference count in the managed object and using boost::intrusive_ptr instead. I have come up with an implementation that I would like to post to this list for two reasons. First of all, I would be thrilled if some more knowledgeable folks would look over the code and let me know if my approach is flawed in some way. Secondly, I think the principles I am discussing will be of interest to others on the list, and possibly the code itself. Instead of posting the whole implementation at once, I am going to start out with something very basic. I will then point out some deficiencies and address them one at a time, modifying the code to overcome each one. I think this approach will be more understandable than throwing the whole mess of code out there at once. The attached mod1.cpp file defines a base class for reference counted objects (called "Pointee") and provides the functions required for managing Pointee objects with intrusive_ptr. The module also defines two test classes that are exposed to python. The VerbosePointee class is just a subclass of Pointee that contains print statements for illustration purposes. RefHolder is a subclass of VerbosePointee that contains a member called 'ref', which is an intrusive_ptr that can reference another VerbosePointee object. Here is a sample python session using this module: >>> from mod1 import * >>> r = RefHolder() creating 0x1d544af0 >>> print r.ref None >>> v = VerbosePointee() creating 0x1d6b6190 >>> r.ref = v >>> print r.ref >>> r.ref is v False >>> r.ref.id == v.id True >>> del v >>> del r destroying 0x1d6b6190 destroying 0x1d544af0 >>> r1 = RefHolder() creating 0x1de86a80 >>> r2 = RefHolder() creating 0x1d544bf0 >>> r1.ref = r2 -------------------------- Boost.Python.ArgumentError ArgumentError: Python argument types in None.None(RefHolder, RefHolder) did not match C++ signature: None(RefHolder {lvalue}, boost::intrusive_ptr) >>> r1.ref = None --------------------------------------------------------------------------- Boost.Python.ArgumentError ArgumentError: Python argument types in None.None(RefHolder, NoneType) did not match C++ signature: None(RefHolder {lvalue}, boost::intrusive_ptr) The memory management works as one would expect, but a few deficiencies are evident in the session shown above: (1) There is an object identity problem (i.e. 'r.ref is v' is False). This means that there are two python wrappers for a single C++ object. In some cases this is okay, but it is not ideal. (2) RefHolder objects cannot hold a reference to another RefHolder object, even though RefHolder is a subclass of VerbosePointee. (3) Although a null pointer is translated to python as None, a RefHolder's ref member cannot be assigned a value of None. In future posts, I will post my proposed solution to the problems mentioned above and address other issues as they arise. Any comments/questions so far? -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: mod1.cpp Type: text/x-c++src Size: 2816 bytes Desc: not available URL: From rwgk at yahoo.com Thu Jul 24 23:14:35 2008 From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve) Date: Thu, 24 Jul 2008 14:14:35 -0700 (PDT) Subject: [C++-sig] Wrapping std::set using Boost::Python Message-ID: <101121.96354.qm@web31108.mail.mud.yahoo.com> You can easily do it from scratch (as an alternative to the indexing suite): http://phenix-online.org/cctbx_sources/scitbx/include/scitbx/stl/set_wrapper.h The sources are also here: http://cctbx.svn.sourceforge.net/viewvc/cctbx/trunk/ but SourceForge viewvc is currently down. Ralf ----- Original Message ---- From: Eric First To: C++-sig at python.org Sent: Thursday, July 24, 2008 7:40:52 AM Subject: [C++-sig] Wrapping std::set using Boost::Python Has anybody wrapped std::set using Boost::Python? I'm looking for the most efficient way to do this. All ideas are greatly appreciated. Thanks. From rogeeff at gmail.com Fri Jul 25 04:13:17 2008 From: rogeeff at gmail.com (Gennadiy Rozental) Date: Fri, 25 Jul 2008 02:13:17 +0000 (UTC) Subject: [C++-sig] using boost::python::list .... References: <37c1dd070807241249l3f6a6a43g727c3a7ff9e100fe@mail.gmail.com> Message-ID: Eric Texier gmail.com> writes: > To make it short: > - should I use the way I do "position" (dynamic) or the way I do for "lookat"(static).? Later, I think. Just create bp::list and assign it. It keeps ref count internally and is cheap to copy (I think). Gennadiy From mathieu.malaterre at gmail.com Fri Jul 25 10:30:33 2008 From: mathieu.malaterre at gmail.com (Mathieu Malaterre) Date: Fri, 25 Jul 2008 10:30:33 +0200 Subject: [C++-sig] Wrapping std::set using Boost::Python In-Reply-To: <101121.96354.qm@web31108.mail.mud.yahoo.com> References: <101121.96354.qm@web31108.mail.mud.yahoo.com> Message-ID: Ralf, apparently viewvc just went back on. During the transition period with no viewvc, I would send path using the svnroot, eg: http://cctbx.svn.sourceforge.net/svnroot/cctbx/trunk/ no history, but at least people can read the last rev of the files. 2cts -Mathieu On Thu, Jul 24, 2008 at 11:14 PM, Ralf W. Grosse-Kunstleve wrote: > You can easily do it from scratch (as an alternative to the indexing suite): > > http://phenix-online.org/cctbx_sources/scitbx/include/scitbx/stl/set_wrapper.h > > The sources are also here: > http://cctbx.svn.sourceforge.net/viewvc/cctbx/trunk/ > but SourceForge viewvc is currently down. > > Ralf > > > ----- Original Message ---- > From: Eric First > To: C++-sig at python.org > Sent: Thursday, July 24, 2008 7:40:52 AM > Subject: [C++-sig] Wrapping std::set using Boost::Python > > > Has anybody wrapped std::set using Boost::Python? I'm looking for the most efficient way to do this. All ideas are greatly appreciated. Thanks. > > _______________________________________________ > C++-sig mailing list > C++-sig at python.org > http://mail.python.org/mailman/listinfo/c++-sig > -- Success is getting what you want. Happiness is wanting what you get. From jeff.webb at nta-inc.net Fri Jul 25 20:29:02 2008 From: jeff.webb at nta-inc.net (Jeff Webb) Date: Fri, 25 Jul 2008 13:29:02 -0500 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <4888E4AE.3000500@nta-inc.net> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> Message-ID: <488A1B6E.3030905@nta-inc.net> At the end of my last post, I listed several deficiencies in my proposed python wrapper, mod1.cpp. Let me start by addressing item (2): > (2) RefHolder objects cannot hold a reference to another RefHolder > object, even though RefHolder is a subclass of VerbosePointee. A simple solution to this problem is to add the following statement at the end of BOOST_PYTHON_MODULE(mod1): implicitly_convertible, intrusive_ptr >(); This statement tells boost::python about the relationship between intrusive_ptr and intrusive_ptr and enables a python user to pass wrapped RefHolder objects to the RefHolder::set_ref function, which is expecting an intrusive_ptr. Let's see if this actually works: >>> from mod1 import * >>> r1 = RefHolder() creating 0xe49c6e0 >>> r2 = RefHolder() creating 0xda13bb0 >>> r1.ref = r2 >>> type(r1.ref) So far, so good. Now on to the next problem: > (3) Although a null pointer is translated to python as None, a > RefHolder's ref member cannot be assigned a value of None. Let's make sure adding the implicitly_convertible call didn't somehow fix the problem: >>> r1.ref = None --------------------------------------------------------------------------- Boost.Python.ArgumentError Traceback (most recent call last) ArgumentError: Python argument types in None.None(RefHolder, NoneType) did not match C++ signature: None(RefHolder {lvalue}, boost::intrusive_ptr) As you can see, this is still an issue. To find a solution, I looked at how the problem was solved for shared_ptr. The solution can be found in this file: include/boost/python/converter/shared_ptr_from_python.hpp I basically made a copy of this function, named it 'intrusive_ptr_from_python', replaced shared_ptr with intrusive_ptr, and left the shared_ptr_deleter argument off of the call to the pointer constructor. In order to register this conversion, the function template must be called from BOOST_PYTHON_MODULE(mod1): boost::python::converter::intrusive_ptr_from_python(); As it turns out, this converter also handles the implicit conversion discussed above, so the implicitly_convertible call is no longer necessary. Now let's test out our new changes, as shown in the attached file, mod2.cpp. >>> from mod2 import * >>> r1 = RefHolder() creating 0xd78af0 >>> r2 = RefHolder() creating 0x17ff260 >>> r1.ref = r2 >>> r1.ref is r2 False >>> r1.ref.id == r2.id True >>> type(r1.ref) >>> v = VerbosePointee() creating 0x1817370 >>> r2.ref = v >>> del r2 >>> r1.ref.ref.id == v.id True >>> del v >>> r1.ref.ref = None destroying 0x1817370 >>> del r1 destroying 0x17ff260 destroying 0xd78af0 So, it looks like this approach handles two of our issues. The object identity problem still remains, but I will save that discussion for another post. Any questions or comments are welcome. -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: mod2.cpp Type: text/x-c++src Size: 4506 bytes Desc: not available URL: From tbourden at doc.ic.ac.uk Sat Jul 26 13:59:22 2008 From: tbourden at doc.ic.ac.uk (Themistoklis Bourdenas) Date: Sat, 26 Jul 2008 12:59:22 +0100 Subject: [C++-sig] wrapping the PyObject_Call() interface Message-ID: <7fd737460807260459n7ce87771r41b3854ece7cff21@mail.gmail.com> Hi, I was looking in the boost.python for a wrapper of the PyObject_Call() func but couldn't find anything. Did I missed it or has it not been implemented yet? In essence I need to call functions in python by providing a collection (either stl or boost.python collection) of boost::python::object instances instead of giving the arguments in the traditional way. Any ideas, or should I use directly PyObject_Call()? Cheers, Themis -------------- next part -------------- An HTML attachment was scrubbed... URL: From seefeld at sympatico.ca Sat Jul 26 14:46:27 2008 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Sat, 26 Jul 2008 08:46:27 -0400 Subject: [C++-sig] wrapping the PyObject_Call() interface In-Reply-To: <7fd737460807260459n7ce87771r41b3854ece7cff21@mail.gmail.com> References: <7fd737460807260459n7ce87771r41b3854ece7cff21@mail.gmail.com> Message-ID: <488B1CA3.5000304@sympatico.ca> Themistoklis Bourdenas wrote: > Hi, I was looking in the boost.python for a wrapper of the > PyObject_Call() func but couldn't find anything. Did I missed it or > has it not been implemented yet? In essence I need to call functions > in python by providing a collection (either stl or boost.python > collection) of boost::python::object instances instead of giving the > arguments in the traditional way. > > Any ideas, or should I use directly PyObject_Call()? I have a patch for this (https://svn.boost.org/trac/boost/ticket/1316), but it requires a final review. You may apply it to your sources locally. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin... From borpat at seznam.cz Sat Jul 26 16:51:51 2008 From: borpat at seznam.cz (Borek Patzak) Date: Sat, 26 Jul 2008 16:51:51 +0200 Subject: [C++-sig] Overloaded functions returning Ll and r values Message-ID: <488B3A07.4000503@seznam.cz> Hi, I have a question concerning wrapping overloaded functions returning both l-value and r-value. The simple example: class Array { int size; double *values; Array (int size) {this->size = size; values = new double[size]; } Array () {size=0; values=NULL;} ~Array () {if (size) delete[] values;} inline double &at(int i) { return values [ i - 1 ]; } inline double at(int i) const { return values [ i - 1 ]; } }; double (Array::*at1)(int) const = &Array::at; double& (Array::*at2)(int) = &Array::at; BOOST_PYTHON_MODULE (array_ext) { class_("Array") .def(init()) // the following line generates an error, when invoking bjam .def("at", at2, return_internal_reference<>()) .def("get", at1) ; } I am unable to generate wrapper for "double& Array ::at(int i)". Any help? Best regards, Borek From seefeld at sympatico.ca Sat Jul 26 17:12:18 2008 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Sat, 26 Jul 2008 11:12:18 -0400 Subject: [C++-sig] Overloaded functions returning Ll and r values In-Reply-To: <488B3A07.4000503@seznam.cz> References: <488B3A07.4000503@seznam.cz> Message-ID: <488B3ED2.8000704@sympatico.ca> Borek Patzak wrote: > > I am unable to generate wrapper for "double& Array ::at(int i)". > Any help? > You can't return or pass float, int, string, etc. by reference in Python, as they are immutable. HTH, Stefan -- ...ich hab' noch einen Koffer in Berlin... From bourdenas at gmail.com Sat Jul 26 19:07:30 2008 From: bourdenas at gmail.com (Themis Bourdenas) Date: Sat, 26 Jul 2008 18:07:30 +0100 Subject: [C++-sig] wrapping the PyObject_Call() interface In-Reply-To: <488B1CA3.5000304@sympatico.ca> References: <7fd737460807260459n7ce87771r41b3854ece7cff21@mail.gmail.com> <488B1CA3.5000304@sympatico.ca> Message-ID: <7fd737460807261007l18624d23vc99bd009b5bbc658@mail.gmail.com> Thanks that works fine. Though I'd like to ask you whether there is any reason for unary operator * to work only with boost::python::tuple and not with boost::python::list. At first I used the latter and I got a python exception that while the functions I was calling was requiring N arguments, it was provided with 0. It was also the case of a bound member function, meaning that at least the self argument was already embedded in the boost::python::object I was calling. When I converted the list to a tuple the call run smoothly. Is this patch going to be in the 1.36.0 release? If not are there any plans for which release it will be included in? Cheers, Themis On Sat, Jul 26, 2008 at 1:46 PM, Stefan Seefeld wrote: > Themistoklis Bourdenas wrote: > >> Hi, I was looking in the boost.python for a wrapper of the PyObject_Call() >> func but couldn't find anything. Did I missed it or has it not been >> implemented yet? In essence I need to call functions in python by providing >> a collection (either stl or boost.python collection) of >> boost::python::object instances instead of giving the arguments in the >> traditional way. >> >> Any ideas, or should I use directly PyObject_Call()? >> > I have a patch for this (https://svn.boost.org/trac/boost/ticket/1316), > but it requires a final review. You may apply it to your sources locally. > > Regards, > Stefan > > -- > > ...ich hab' noch einen Koffer in Berlin... > > _______________________________________________ > C++-sig mailing list > C++-sig at python.org > http://mail.python.org/mailman/listinfo/c++-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at boostpro.com Sat Jul 26 19:55:59 2008 From: dave at boostpro.com (David Abrahams) Date: Sat, 26 Jul 2008 13:55:59 -0400 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <4888E4AE.3000500@nta-inc.net> (Jeff Webb's message of "Thu, 24 Jul 2008 15:23:10 -0500") References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> Message-ID: <87iqusv8b4.fsf@mcbain.luannocracy.com> on Thu Jul 24 2008, Jeff Webb wrote: > The boost::python library provides excellent support for wrapping C++ > objects managed by boost::shared_ptr reference counting smart > pointers. I am attempting to achieve similar functionality with a > little less memory overhead by embedding the reference count in the > managed object and using boost::intrusive_ptr instead. Why? Have you profiled the memory use of shared_ptr and found it to be a problem? -- Dave Abrahams BoostPro Computing http://www.boostpro.com From mateuszb at gmail.com Sun Jul 27 03:01:21 2008 From: mateuszb at gmail.com (Mateusz Berezecki) Date: Sun, 27 Jul 2008 03:01:21 +0200 Subject: [C++-sig] boost.python fails to recognize type Message-ID: Hi list readers, I want to enable Python to access the C++ class. I do this as follows: BOOST_PYTHON_MODULE(ht) { <...> class_("ScanSpec") .def_readwrite("row_limit", &Hypertable::ScanSpec::row_limit) .def_readwrite("max_versions", &Hypertable::ScanSpec::max_versions) .def_readwrite("columns", &Hypertable::ScanSpec::columns) .def_readwrite("start_row", &Hypertable::ScanSpec::start_row) .def_readwrite("start_row_inclusive", &Hypertable::ScanSpec::start_row_inclusive) .def_readwrite("end_row", &Hypertable::ScanSpec::end_row) .def_readwrite("end_row_inclusive", &Hypertable::ScanSpec::end_row_inclusive) ; <...> } The ScanSpec class is defined as follows: 00087 class ScanSpec { 00088 public: 00089 ScanSpec(); 00090 size_t encoded_length(); 00091 void encode(uint8_t **bufPtr); 00092 bool decode(uint8_t **bufPtr, size_t *remainingPtr); 00093 uint32_t row_limit; 00094 uint32_t max_versions; 00095 std::vector columns; 00096 const char *start_row; 00097 bool start_row_inclusive; 00098 const char *end_row; 00099 bool end_row_inclusive; 00100 std::pair interval; 00101 bool return_deletes; 00102 }; The conversion process happens in the Table class wrapper which is defined as follows class Table { private: Hypertable::TablePtr m_table_ptr; public: Table() {} Table(Hypertable::TablePtr ptr) : m_table_ptr(ptr) {} TableScanner create_scanner(object pyspec) { Hypertable::ScanSpec& spec = extract(pyspec); return TableScanner(m_table_ptr->create_scanner(spec, 10)); } }; The error is as follows TypeError: No registered converter was able to extract a C++ reference to type Hypertable::ScanSpec from this Python object of type Boost.Python.class I've also tried extracting it as a pointer TypeError: No registered converter was able to extract a C++ pointer to type Hypertable::ScanSpec from this Python object of type Boost.Python.class And the same happens when trying to extract a copy of ScanSpec, but this time the error message tells it's not possible to extract rvalue from Boost.Python.class. I've tried every possible combination and it doesn't work. Can somebody please help me ? Mateusz From seefeld at sympatico.ca Sun Jul 27 03:29:41 2008 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Sat, 26 Jul 2008 21:29:41 -0400 Subject: [C++-sig] wrapping the PyObject_Call() interface In-Reply-To: <7fd737460807261007l18624d23vc99bd009b5bbc658@mail.gmail.com> References: <7fd737460807260459n7ce87771r41b3854ece7cff21@mail.gmail.com> <488B1CA3.5000304@sympatico.ca> <7fd737460807261007l18624d23vc99bd009b5bbc658@mail.gmail.com> Message-ID: <488BCF85.4070204@sympatico.ca> Themis Bourdenas wrote: > Thanks that works fine. > > Though I'd like to ask you whether there is any reason for unary > operator * to work only with boost::python::tuple and not with > boost::python::list. At first I used the latter and I got a python > exception that while the functions I was calling was requiring N > arguments, it was provided with 0. It was also the case of a bound > member function, meaning that at least the self argument was already > embedded in the boost::python::object I was calling. When I converted > the list to a tuple the call run smoothly. My patch itself is agnostic to the actual types you pass for args and kwds. The requirement may stem from Python's runtime itself, I'm not sure (and I have no time to test right now), though I find that a bit surprising, as it should be polymorphic (it could even be a user-defined type implementing the sequence protocol). > Is this patch going to be in the 1.36.0 release? If not are there any > plans for which release it will be included in? The patch is going to be checked into trunk, and will end up in whatever releases branch from that. Unfortunately it's too late for the 36 release. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin... From gjcarneiro at gmail.com Sun Jul 27 19:56:37 2008 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Sun, 27 Jul 2008 18:56:37 +0100 Subject: [C++-sig] Announce: PyBindGen 0.9 released Message-ID: PyBindGen is a Python module that is geared to generating C/C++ code that binds a C/C++ library for Python. It does so without extensive use of either C++ templates or C pre-processor macros. It has modular handling of C/C++ types, and can be easily extended with Python plugins. The generated code is almost as clean as what a human programmer would write. It can be downloaded from: http://code.google.com/p/pybindgen/ Bug reports should be filed here: https://bugs.launchpad.net/pybindgen Special thanks to Roman Yakovenko (lots of help with pygccxml) and Mathieu Lacage (API suggestions, documentation writing). === pybindgen 0.9 === - Fix GCC 4.2 compilation warnings; - Works with some GCCXML 0.9/cvs snapshots (tested with 2008-04-20), in addition to stable 0.7; - Support for overloaded virtual methods; - Add 'ignore' annotation support, allowing to ignore functions and methods; - Generally work hard to make sure the generated code at least always compiles, even if we have to disable generation of certain wrappers; - Add support for protected methods and constructors; - Preliminary support for templated classes/methods/functions; - Add more type conversions, such as uint64_t and uint16_t; - Support implicit conversions also for pass-by-reference parameters; - Add supported for nested (i.e. defined inside a class) enums and classes; - Add support for adding manually written custom method or function wrappers; - Split the gccxmlparser.ModuleParser.parse() method into several smaller methods, to allow greater customization and flexibility; - Add support for customising C++ class instance creation code; - Much improved support for wrapping pure C code; - Support std::ostream << myobject mapped as str(myobject) (Mathieu Lacage) - Support default values in parameters; - More intuitive API (thanks Mathieu Lacage for feedback) - Support generation of a Python pybindgen script from scanned API; - Support splitting of generated python script and/or C/C++ module into several files; - Lots of small bug fixes, and other features I probably forgot; - New tutorial (thanks Mathieu Lacage), and API docs. -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From roman.yakovenko at gmail.com Sun Jul 27 20:33:07 2008 From: roman.yakovenko at gmail.com (Roman Yakovenko) Date: Sun, 27 Jul 2008 21:33:07 +0300 Subject: [C++-sig] boost.python fails to recognize type In-Reply-To: References: Message-ID: <7465b6170807271133k5ee7512fv76c8cca2ee7b698c@mail.gmail.com> On Sun, Jul 27, 2008 at 4:01 AM, Mateusz Berezecki wrote: > Hi list readers, > > I want to enable Python to access the C++ class. > I do this as follows: > > BOOST_PYTHON_MODULE(ht) > { > <...> > class_("ScanSpec") > .def_readwrite("row_limit", &Hypertable::ScanSpec::row_limit) > .def_readwrite("max_versions", &Hypertable::ScanSpec::max_versions) > .def_readwrite("columns", &Hypertable::ScanSpec::columns) > .def_readwrite("start_row", &Hypertable::ScanSpec::start_row) > .def_readwrite("start_row_inclusive", > &Hypertable::ScanSpec::start_row_inclusive) > .def_readwrite("end_row", &Hypertable::ScanSpec::end_row) > .def_readwrite("end_row_inclusive", > &Hypertable::ScanSpec::end_row_inclusive) > ; > <...> > } > > The ScanSpec class is defined as follows: > 00087 class ScanSpec { > 00088 public: > 00089 ScanSpec(); > 00090 size_t encoded_length(); > 00091 void encode(uint8_t **bufPtr); > 00092 bool decode(uint8_t **bufPtr, size_t *remainingPtr); > 00093 uint32_t row_limit; > 00094 uint32_t max_versions; > 00095 std::vector columns; > 00096 const char *start_row; > 00097 bool start_row_inclusive; > 00098 const char *end_row; > 00099 bool end_row_inclusive; > 00100 std::pair interval; > 00101 bool return_deletes; > 00102 }; > > The conversion process happens in the Table class wrapper which is > defined as follows > class Table { > private: > Hypertable::TablePtr m_table_ptr; > > public: > Table() {} > Table(Hypertable::TablePtr ptr) : m_table_ptr(ptr) {} > > TableScanner create_scanner(object pyspec) > { > Hypertable::ScanSpec& spec = extract(pyspec); > return TableScanner(m_table_ptr->create_scanner(spec, 10)); > } > }; > > The error is as follows > TypeError: No registered converter was able to extract a C++ reference > to type Hypertable::ScanSpec from this Python object of type > Boost.Python.class > > I've also tried extracting it as a pointer > TypeError: No registered converter was able to extract a C++ pointer > to type Hypertable::ScanSpec from this Python object of type > Boost.Python.class > > And the same happens when trying to extract a copy of ScanSpec, but > this time the error message tells it's not possible to extract rvalue > from Boost.Python.class. > > I've tried every possible combination and it doesn't work. > > Can somebody please help me ? The first step is to create small and complete example, that shows your problem. -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/ From rogeeff at gmail.com Sun Jul 27 20:45:32 2008 From: rogeeff at gmail.com (Gennadiy Rozental) Date: Sun, 27 Jul 2008 18:45:32 +0000 (UTC) Subject: [C++-sig] boost.python fails to recognize type References: Message-ID: Mateusz Berezecki gmail.com> writes: > TableScanner create_scanner(object pyspec) > { > Hypertable::ScanSpec& spec = extract(pyspec); > return TableScanner(m_table_ptr->create_scanner(spec, 10)); > } > }; > > The error is as follows > TypeError: No registered converter was able to extract a C++ reference > to type Hypertable::ScanSpec from this Python object of type > Boost.Python.class Obviuosly you are getting object representing class instead of one representing instance of a class. you did not show how create_scanner is exported/invoked. Maybe you exported it as static method? Gennadiy From jeff.webb at nta-inc.net Mon Jul 28 17:53:34 2008 From: jeff.webb at nta-inc.net (Jeff Webb) Date: Mon, 28 Jul 2008 10:53:34 -0500 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <87iqusv8b4.fsf@mcbain.luannocracy.com> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <87iqusv8b4.fsf@mcbain.luannocracy.com> Message-ID: <488DEB7E.5030108@nta-inc.net> David Abrahams wrote: > on Thu Jul 24 2008, Jeff Webb wrote: > >> The boost::python library provides excellent support for wrapping C++ >> objects managed by boost::shared_ptr reference counting smart >> pointers. I am attempting to achieve similar functionality with a >> little less memory overhead by embedding the reference count in the >> managed object and using boost::intrusive_ptr instead. > > Why? Have you profiled the memory use of shared_ptr and found it to be > a problem? I investigated the memory usage of shared_ptr by running valgrind on a test program and by looking at the shared_ptr source code. My findings are summarized in this post: http://mail.python.org/pipermail/c++-sig/2008-July/013786.html I am designing a C++ library from the ground up, so I am not tied to any existing conventions (base classes, memory management schemes, etc.). I want to be able to expose the fundamental building blocks of my library to python, but I want to allow the library to be used without python as well. I have decided to use smart pointers internally in my C++ library to lay the foundation for seamless integration with python. Some of my library building blocks are very small, and users will most likely have a large number of instances, so the overhead of the internal memory management scheme is a concern. I am less concerned by the overhead of the python wrappers, since I expect a small percentage of these objects to by wrapped by python at any given time. It think that by using intrusive_ptr, I can save one raw pointer's worth of memory for every smart pointer I use, and another raw pointer's worth of memory (plus maybe an int) for the managed objects themselves. I have included the highlights of my previous email below, but you can use the link above if you want to look at the full post. I would be interested in hearing any thoughts you may have on any of this. Thanks, Jeff -------- Original Message -------- Date: Thu, 17 Jul 2008 18:02:26 -0500 From: Jeff Webb To: Development of Python/C++ integration Subject: shared_ptr vs. intrusive_ptr [...] I have attempted to quantify the memory usage of shared_ptr vs. a simple intrusive_ptr implementation using the attached test program. I used valgrind to verify the dynamic memory usage. Here are the results: shared_ptr ---------- Each shared_ptr is the size of two raw pointers (one for the managed object, and one for the shared reference count): A total of 8 bytes on x86 or 16 bytes on x86_64 For each object that is managed by shared_ptrs, a shared reference count must be allocated. The memory usage for the shared reference count appears to be two integers and two pointers: A total of 16 bytes on x86 or 24 bytes on x86_64 When looking at the shared_ptr source code, I could only find one pointer in the shared reference count, but I cannot reconcile this with the memory usage reported by valgrind. I suppose it doesn't really matter, but the breakdown I found in the code is listed below. Maybe I am not looking at the right implementation for my system. sp_counted_impl_p: sp_counted_base: int use_count_; (4 bytes) int weak_count_; (4 bytes) vtable pointer? (4 or 8 bytes) X * (4 or 8 bytes) intrusive_ptr ------------- Each intrusive_ptr is the size of a raw pointer: A total of 4 bytes on x86 or 8 bytes on x86_64 For each object that is being managed by this intrusive_ptr implementation, an extra integer must be added to the object to hold the reference count: A total of 4 bytes on x86 or x86_64 [...] From dave at boostpro.com Mon Jul 28 18:10:58 2008 From: dave at boostpro.com (David Abrahams) Date: Mon, 28 Jul 2008 12:10:58 -0400 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488A1B6E.3030905@nta-inc.net> Message-ID: <87ej5eyood.fsf@mcbain.luannocracy.com> on Fri Jul 25 2008, Jeff Webb wrote: > So, it looks like this approach handles two of our issues. The object > identity problem still remains, but I will save that discussion for > another post. Any questions or comments are welcome. The object identity issue is the big one. Without a custom deleter -- which is why shared_ptr works so smoothly -- there's little you can do about it, AFAIK. -- Dave Abrahams BoostPro Computing http://www.boostpro.com From jeff.webb at nta-inc.net Mon Jul 28 18:39:17 2008 From: jeff.webb at nta-inc.net (Jeff Webb) Date: Mon, 28 Jul 2008 11:39:17 -0500 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <87ej5eyood.fsf@mcbain.luannocracy.com> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488A1B6E.3030905@nta-inc.net> <87ej5eyood.fsf@mcbain.luannocracy.com> Message-ID: <488DF635.4070108@nta-inc.net> David Abrahams wrote: > on Fri Jul 25 2008, Jeff Webb wrote: > >> So, it looks like this approach handles two of our issues. The object >> identity problem still remains, but I will save that discussion for >> another post. Any questions or comments are welcome. > > The object identity issue is the big one. Without a custom deleter -- > which is why shared_ptr works so smoothly -- there's little you can do > about it, AFAIK. Thanks, Dave. I have found a solution that seems to work for me. I will post it soon. I would definitely like to have you and some others take a look at it and see if it looks reasonable. -Jeff From hans_meine at gmx.net Mon Jul 28 19:59:05 2008 From: hans_meine at gmx.net (Hans Meine) Date: Mon, 28 Jul 2008 19:59:05 +0200 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <488DEB7E.5030108@nta-inc.net> References: <87iqusv8b4.fsf@mcbain.luannocracy.com> <488DEB7E.5030108@nta-inc.net> Message-ID: <200807281959.05497.hans_meine@gmx.net> On Montag 28 Juli 2008, Jeff Webb wrote: > I have included the highlights of my previous email below, but you can use > the link above if you want to look at the full post. I would be interested > in hearing any thoughts you may have on any of this. Let me just say thanks for your first two write-ups, they're much appreciated and will most probably be of help for future BPL users with similar needs. -- Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From amohr at pixar.com Mon Jul 28 20:46:36 2008 From: amohr at pixar.com (Alex Mohr) Date: Mon, 28 Jul 2008 11:46:36 -0700 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <87ej5eyood.fsf@mcbain.luannocracy.com> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488A1B6E.3030905@nta-inc.net> <87ej5eyood.fsf@mcbain.luannocracy.com> Message-ID: <488E140C.2020805@pixar.com> > The object identity issue is the big one. Without a custom deleter -- > which is why shared_ptr works so smoothly -- there's little you can do > about it, AFAIK. I have a solution to object identity using custom smart pointers, but it's quite ugly. The crux of it is to register your own to and from python conversions that do the identity work. I posted a "sketch" of how I do this awhile ago: http://mail.python.org/pipermail/c++-sig/2008-January/013265.html Alex From jeff.webb at nta-inc.net Mon Jul 28 23:33:21 2008 From: jeff.webb at nta-inc.net (Jeff Webb) Date: Mon, 28 Jul 2008 16:33:21 -0500 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <488E140C.2020805@pixar.com> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488A1B6E.3030905@nta-inc.net> <87ej5eyood.fsf@mcbain.luannocracy.com> <488E140C.2020805@pixar.com> Message-ID: <488E3B21.5020300@nta-inc.net> Alex Mohr wrote: >> The object identity issue is the big one. Without a custom deleter -- >> which is why shared_ptr works so smoothly -- there's little you can do >> about it, AFAIK. > > I have a solution to object identity using custom smart pointers, but > it's quite ugly. The crux of it is to register your own to and from > python conversions that do the identity work. I posted a "sketch" of > how I do this awhile ago: > > http://mail.python.org/pipermail/c++-sig/2008-January/013265.html Good stuff, Alex. I think I can learn some things from this code. Thanks for posting it and documenting it with comments. -Jeff From mateuszb at gmail.com Tue Jul 29 00:21:46 2008 From: mateuszb at gmail.com (Mateusz Berezecki) Date: Tue, 29 Jul 2008 00:21:46 +0200 Subject: [C++-sig] boost.python fails to recognize type Message-ID: Hello, That's the complete example code #include #include #include #include #include #include #include #include #include #include #include #include using namespace boost::python; using Hypertable::ScanSpec; class TableScanner { private: Hypertable::TableScannerPtr m_scanner; public: TableScanner() {} TableScanner(Hypertable::TableScannerPtr ptr) : m_scanner(ptr) { } bool next(Hypertable::Cell& cell) { return m_scanner->next(cell); } }; class Table { private: Hypertable::TablePtr m_table_ptr; public: Table() { } Table(Hypertable::TablePtr ptr) : m_table_ptr(ptr) { } TableScanner create_scanner(object const &pyspec) { extract ex(pyspec); ScanSpec& spec = ex(); return TableScanner(m_table_ptr->create_scanner(spec, 10)); } }; class Client { private: Hypertable::ClientPtr m_client; public: Client(const std::string& argv0) { m_client = new Hypertable::Client(argv0.c_str()); } Client(const std::string& argv0, const std::string& config) { m_client = new Hypertable::Client(argv0.c_str(), config); } Table open_table(const std::string& name) { Hypertable::TablePtr t = m_client->open_table(name); return Table(t); } }; BOOST_PYTHON_MODULE(ht) { class_("Client", init()) .def(init()) .def("open_table", &Client::open_table) ; class_("Table", "Table representation") .def("create_scanner", &Table::create_scanner) ; class_("TableScanner") .def("next", &TableScanner::next) ; class_("ScanSpec", "scan spec docstring") .def_readwrite("row_limit", &ScanSpec::row_limit) .def_readwrite("max_versions", &ScanSpec::max_versions) .def_readwrite("columns", &ScanSpec::columns) .def_readwrite("start_row", &ScanSpec::start_row) .def_readwrite("start_row_inclusive", &ScanSpec::start_row_inclusive) .def_readwrite("end_row", &ScanSpec::end_row) .def_readwrite("end_row_inclusive", &ScanSpec::end_row_inclusive) ; class_("Cell") .def_readonly("row_key", &Hypertable::Cell::row_key) .def_readonly("column_family", &Hypertable::Cell::column_family) .def_readonly("column_qualifier", &Hypertable::Cell::column_qualifier) .def_readonly("timestamp", &Hypertable::Cell::timestamp) .def_readonly("value", &Hypertable::Cell::value) .def_readonly("value_len", &Hypertable::Cell::value_len) .def_readonly("flag", &Hypertable::Cell::flag) ; } and the python code invoking it is: import ht client = ht.Client("ht", "/opt/hypertable/0.9.0.7/conf/hypertable.cfg") table = client.open_table("METADATA") scan_spec = ht.ScanSpec scan_spec.start_row = 0 scan_spec.start_row = "\x00" scan_spec.end_row = "\xff\xff" scan_spec.start_row_inclusive = True scan_spec.end_row_inclusive = True scan_spec.max_versions = 1 print scan_spec.__doc__ print table.__doc__ print scan_spec scanner = table.create_scanner(scan_spec) This yields the errors mentioned in the previous post. I am stuck on this one really badly. I just started playing with Boost.Python 2 days ago. Mateusz From jeff.webb at nta-inc.net Tue Jul 29 00:27:27 2008 From: jeff.webb at nta-inc.net (Jeff Webb) Date: Mon, 28 Jul 2008 17:27:27 -0500 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <4888E4AE.3000500@nta-inc.net> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> Message-ID: <488E47CF.4080803@nta-inc.net> Jeff Webb wrote: > (1) There is an object identity problem (i.e. 'r.ref is v' is False). > This means that there are two python wrappers for a single C++ object. > In some cases this is okay, but it is not ideal. In order to solve the object identity problem mentioned above, I needed some method of storing the PyObject pointer that is wrapping a given Pointee object and a way of looking it up. There are two methods that I have tried: (1) storing the PyObject pointer in the Pointee object itself (2) using a global dictionary to store the Pointee->PyObject mappings I initially used solution (1), which worked well for me since I have control over the Pointee data structure. I later decided that I would rather not have the overhead of an extra pointer stored in every Pointee object, whether it was accessed from python or not, so I tried approach (2), which is what I am posting here. If you like method (1) better, I think it will be easy for you to modify the attached example to work that way. I have modified my previous example (mod2.cpp) in several ways: (1) I added set_pyobject_for_pointee, get_pyobject_for_pointee, and clear_pyobject_for_pointee functions that manipulate a global std::map object. (2) I have specialized the make_instance_impl template for a T object held by an intrusive_ptr. (3) I added an init_class function template along with a helper function called install_pointer_in_pyobject. (4) I have modified the class wrappers by adding a no_init argument to the class_ statements, adding an __init__ method to call the init_class function, and adding a __del__ method that removes the pyobject pointer for the corresponding C++ object when a python wrapper is destroyed. The make_instance_impl template is where code was added to check for an existing python wrapper and to record the PyObject pointer when a new wrapper is generated. The init_class method is necessary to record the PyObject pointer when an object is created from python because the make_instance_impl template is not called in that case. The __del__ method is necessary to remove a PyObject pointer from the std::map object when a python wrapper is destroyed. I'm sure there are better ways to do some of these things, but I think you will find my modifications understandable. It see at least two main areas where there could be memory leaks: the install_pointer_in_pyobject function and the code in make_instance_impl that returns a reference to an existing pyobject. If someone could at least take a look at those sections, I would appreciate it. Here is a sample session with the new version of our module: >>> from mod3 import * >>> r1 = RefHolder() creating 0xb0741c0 >>> r2 = RefHolder() creating 0xb077140 >>> r1.ref = r2 >>> r1.ref is r2 True >>> v = VerbosePointee() creating 0xb9a95f0 >>> r2.ref = v >>> del v >>> del r2 >>> print r1.ref >>> r1.ref.ref = None destroying 0xb9a95f0 >>> del r1 destroying 0xb077140 destroying 0xb0741c0 >From the output above, it appears that the object identity problem I mentioned has been solved. The __del__ method also seems to have done it's job since we can still access r1.ref without a segmentation fault even after r2 has been deleted. This approach has worked reliably as far as I can tell. Any comments or questions on the implementation are more than welcome. -Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: mod3.cpp Type: text/x-c++src Size: 10849 bytes Desc: not available URL: From seefeld at sympatico.ca Tue Jul 29 00:37:51 2008 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Mon, 28 Jul 2008 18:37:51 -0400 Subject: [C++-sig] boost.python fails to recognize type In-Reply-To: References: Message-ID: <488E4A3F.4030502@sympatico.ca> Mateusz Berezecki wrote: > Hello, > > That's the complete example code > I guess you missed the other keyword: small (or as I would have spelled it: minimal). As Gennadi guessed, you didn't instantiate your ScanSpec. Try replacing scan_spec = ht.ScanSpec by scan_spec = ht.ScanSpec() and all should work fine. HTH, Stefan -- ...ich hab' noch einen Koffer in Berlin... From dave at boostpro.com Tue Jul 29 01:23:44 2008 From: dave at boostpro.com (David Abrahams) Date: Mon, 28 Jul 2008 19:23:44 -0400 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <488E47CF.4080803@nta-inc.net> (Jeff Webb's message of "Mon, 28 Jul 2008 17:27:27 -0500") References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488E47CF.4080803@nta-inc.net> Message-ID: <87ej5dy4n3.fsf@mcbain.luannocracy.com> on Mon Jul 28 2008, Jeff Webb wrote: > Jeff Webb wrote: >> (1) There is an object identity problem (i.e. 'r.ref is v' is >> False). This means that there are two python wrappers for a single >> C++ object. In some cases this is okay, but it is not ideal. > > In order to solve the object identity problem mentioned above, I needed some method of storing the PyObject pointer that is wrapping a given Pointee object and a way of looking it up. There are two methods that I have tried: > > (1) storing the PyObject pointer in the Pointee object itself http://www.boost.org/doc/libs/1_35_0/libs/python/doc/v2/has_back_reference.html > (2) using a global dictionary to store the Pointee->PyObject mappings Isn't this going to incur comparable memory use to shared_ptr? -- Dave Abrahams BoostPro Computing http://www.boostpro.com From dave at boostpro.com Tue Jul 29 01:37:58 2008 From: dave at boostpro.com (David Abrahams) Date: Mon, 28 Jul 2008 19:37:58 -0400 Subject: [C++-sig] boost.python fails to recognize type In-Reply-To: (Mateusz Berezecki's message of "Tue, 29 Jul 2008 00:21:46 +0200") References: Message-ID: <87abg1y3zd.fsf@mcbain.luannocracy.com> on Mon Jul 28 2008, "Mateusz Berezecki" wrote: > This yields the errors mentioned in the previous post. But it's huge. Please systematically cut it down to the absolute minimum you need in order to be able to demonstrate the problem. I'll bet you find the source of the problem yourself, but if you don't, we'll be able to help you. -- Dave Abrahams BoostPro Computing http://www.boostpro.com From amohr at pixar.com Tue Jul 29 02:20:02 2008 From: amohr at pixar.com (Alex Mohr) Date: Mon, 28 Jul 2008 17:20:02 -0700 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <87ej5dy4n3.fsf@mcbain.luannocracy.com> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488E47CF.4080803@nta-inc.net> <87ej5dy4n3.fsf@mcbain.luannocracy.com> Message-ID: <488E6232.9030605@pixar.com> >> (2) using a global dictionary to store the Pointee->PyObject mappings > > Isn't this going to incur comparable memory use to shared_ptr? We use a global table. Suppose you have a system where there are hundreds of thousands of objects and only some small fraction of them are passed to python. In this case, adding 8 bytes of overhead to every object versus adding 8 bytes of overhead to only those objects passed to python can save 100s of KB of memory. Perhaps that doesn't sound like a lot these days, but when your users are always trying to push the hardware and software as hard as they possibly can, it can definitely make a difference. Formerly I embedded the python objects inside the C++ objects but switched to the global table due to memory usage. Alex From dave at boostpro.com Tue Jul 29 02:53:14 2008 From: dave at boostpro.com (David Abrahams) Date: Mon, 28 Jul 2008 20:53:14 -0400 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <488E6232.9030605@pixar.com> (Alex Mohr's message of "Mon, 28 Jul 2008 17:20:02 -0700") References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488E47CF.4080803@nta-inc.net> <87ej5dy4n3.fsf@mcbain.luannocracy.com> <488E6232.9030605@pixar.com> Message-ID: <87od4hwlxh.fsf@mcbain.luannocracy.com> on Mon Jul 28 2008, Alex Mohr wrote: >>> (2) using a global dictionary to store the Pointee->PyObject mappings >> >> Isn't this going to incur comparable memory use to shared_ptr? > > We use a global table. > > Suppose you have a system where there are hundreds of thousands of > objects and only some small fraction of them are passed to python. > > In this case, adding 8 bytes of overhead to every object versus adding > 8 bytes of overhead to only those objects passed to python can save > 100s of KB of memory. Perhaps that doesn't sound like a lot these > days, but when your users are always trying to push the hardware and > software as hard as they possibly can, it can definitely make a > difference. > > Formerly I embedded the python objects inside the C++ objects but > switched to the global table due to memory usage. I wasn't talking about adding storage to every instance of a class. Boost.Python converts any wrapped object of type T to a shared_ptr which, when passed back to Python, gets converted back to the original Python object. The last shared_ptr to that object may be reclaimed when the C++ code is done using it. -- Dave Abrahams BoostPro Computing http://www.boostpro.com From jeff.webb at nta-inc.net Tue Jul 29 04:41:16 2008 From: jeff.webb at nta-inc.net (Jeff Webb) Date: Mon, 28 Jul 2008 21:41:16 -0500 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <87od4hwlxh.fsf@mcbain.luannocracy.com> References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488E47CF.4080803@nta-inc.net> <87ej5dy4n3.fsf@mcbain.luannocracy.com> <488E6232.9030605@pixar.com> <87od4hwlxh.fsf@mcbain.luannocracy.com> Message-ID: <488E834C.2030601@nta-inc.net> David Abrahams wrote: > on Mon Jul 28 2008, Alex Mohr wrote: > >>>> (2) using a global dictionary to store the Pointee->PyObject mappings >>> Isn't this going to incur comparable memory use to shared_ptr? >> We use a global table. >> >> Suppose you have a system where there are hundreds of thousands of >> objects and only some small fraction of them are passed to python. >> >> In this case, adding 8 bytes of overhead to every object versus adding >> 8 bytes of overhead to only those objects passed to python can save >> 100s of KB of memory. Perhaps that doesn't sound like a lot these >> days, but when your users are always trying to push the hardware and >> software as hard as they possibly can, it can definitely make a >> difference. >> >> Formerly I embedded the python objects inside the C++ objects but >> switched to the global table due to memory usage. > > I wasn't talking about adding storage to every instance of a class. > > Boost.Python converts any wrapped object of type T to a shared_ptr > which, when passed back to Python, gets converted back to the original > Python object. The last shared_ptr to that object may be reclaimed when > the C++ code is done using it. It sounds like Alex and I have very similar applications, or at least we're thinking along the same lines. I am using smart pointers for memory management within the C++ library itself. Objects hold smart pointers to other objects, and smart pointers to objects are passed as arguments to various member functions. I want these smart pointers to be as memory efficient as possible because there will be a lot of them. I also want to be able to expose any of these objects (as well as functions that take smart pointers to these objects) to python without adding memory overhead to the internal smart pointers or the class instances themselves. As Alex mentioned, adding some overhead to keep track of the wrapped objects is acceptable (and probably unavoidable), so the global table seems like a good option. To clarify, are you suggesting that I should change my internal C++ code to use shared_ptrs everywhere, or something else? Sorry that it's taking us a while to get on the same page. -Jeff From dave at boostpro.com Tue Jul 29 15:47:11 2008 From: dave at boostpro.com (David Abrahams) Date: Tue, 29 Jul 2008 09:47:11 -0400 Subject: [C++-sig] alternative smart pointer (intrusive_ptr) example In-Reply-To: <488E834C.2030601@nta-inc.net> (Jeff Webb's message of "Mon, 28 Jul 2008 21:41:16 -0500") References: <200807220656.22034.meine@informatik.uni-hamburg.de> <4885ED8D.6090804@nta-inc.net> <200807231119.36451.meine@informatik.uni-hamburg.de> <4888E4AE.3000500@nta-inc.net> <488E47CF.4080803@nta-inc.net> <87ej5dy4n3.fsf@mcbain.luannocracy.com> <488E6232.9030605@pixar.com> <87od4hwlxh.fsf@mcbain.luannocracy.com> <488E834C.2030601@nta-inc.net> Message-ID: <87wsj4u7j4.fsf@mcbain.luannocracy.com> on Mon Jul 28 2008, Jeff Webb wrote: > David Abrahams wrote: >> on Mon Jul 28 2008, Alex Mohr wrote: >> >>>>> (2) using a global dictionary to store the Pointee->PyObject mappings >>>> Isn't this going to incur comparable memory use to shared_ptr? >>> We use a global table. >>> >>> Suppose you have a system where there are hundreds of thousands of >>> objects and only some small fraction of them are passed to python. >>> >>> In this case, adding 8 bytes of overhead to every object versus adding >>> 8 bytes of overhead to only those objects passed to python can save >>> 100s of KB of memory. Perhaps that doesn't sound like a lot these >>> days, but when your users are always trying to push the hardware and >>> software as hard as they possibly can, it can definitely make a >>> difference. >>> >>> Formerly I embedded the python objects inside the C++ objects but >>> switched to the global table due to memory usage. >> >> I wasn't talking about adding storage to every instance of a class. >> >> Boost.Python converts any wrapped object of type T to a shared_ptr >> which, when passed back to Python, gets converted back to the original >> Python object. The last shared_ptr to that object may be reclaimed when >> the C++ code is done using it. > > It sounds like Alex and I have very similar applications, or at least > we're thinking along the same lines. I am using smart pointers for > memory management within the C++ library itself. Objects hold smart > pointers to other objects, and smart pointers to objects are passed as > arguments to various member functions. I want these smart pointers to > be as memory efficient as possible because there will be a lot of > them. > > I also want to be able to expose any of these objects (as well as > functions that take smart pointers to these objects) to python without > adding memory overhead to the internal smart pointers or the class > instances themselves. As Alex mentioned, adding some overhead to keep > track of the wrapped objects is acceptable (and probably unavoidable), > so the global table seems like a good option. OK. > To clarify, are you suggesting that I should change my internal C++ > code to use shared_ptrs everywhere, or something else? Nope, I was forgetting that doubtless also want to use your Python-exposed API from C++. -- Dave Abrahams BoostPro Computing http://www.boostpro.com From pwinston at gmail.com Wed Jul 30 05:14:18 2008 From: pwinston at gmail.com (Philip Winston) Date: Tue, 29 Jul 2008 23:14:18 -0400 Subject: [C++-sig] application exit and shared_ptr Message-ID: <33677e5d0807292014s5cae77cdm7fa02b0fc21e6645@mail.gmail.com> I understand it is not possible to call Py_Finalize with Boost.Python. But is it possible to manually delete all variables from an embedded interactive interpreter? Would this be a way to to cause Python to release any shared_ptrs it is holding on to? So we return shared_ptrs from our API. But at exit (and possibly other times) we'd like to reset the interpreter, so it's no longer holding any shared_ptrs, so those objects are free to be deleted if no other references are held. Some specific excerpts (any other advice welcome): excerpts: m_mainmodule = import("__main__"); m_namespace = m_mainmodule.attr("__dict__"); exec("import PyInteractive"); exec("__pyInteractiveExec = PyInteractive.PyInteractive().executeline"); m_interactive = extract