[C++-sig] Re: Problem: held type, std::auto_ptr, ReferenceErrors.
David Abrahams
dave at boost-consulting.com
Tue Aug 26 01:16:40 CEST 2003
Prabhu Ramachandran <prabhu at aero.iitm.ernet.in> writes:
> Hi Dave,
>
> Thank you *very* much for the detailed and patient clarifications!
Sure; you're worth it ;-)
> If it will help other users in the future I will try and summarize
> this discussion and put it together so it can be part of the docs. I
> personally feel that a high level "how it works - with examples" kind
> of write up would make it easier for characters like me to understand
> what is going on without having to dive into the code. If you think
> this is a good idea, I'll try and work on something when I get the
> time.
Yes, please! BTW, I've had several promises of such documentation
efforts but so far nobody has had the fortitude to turn my ramblings
into a real document... so you could distinguish yourself by
following through!
For example, here's something which was supposed to become real
documentation:
http://www.boost-consulting.com/boost/libs/python/doc/internals.html
(it's in your CVS tree).
>>>>>> "DA" == David Abrahams <dave at boost-consulting.com> writes:
>
> >> >> Problem 1:
> >>
> DA> If you pass this through c++filt you'll see that it has a
> DA> Python object of type (Python) B but can't match it to a C++
> DA> function requiring a (C++) B lvalue argument. That can only
> DA> mean that there's no C++ B object in the Python B object.
> DA> Given that you're holding these things by auto_ptr, which
> DA> gives up ownership when copied, it's a strong hint that the
> DA> object has been passed where a C++ std::auto_ptr object was
> DA> expected, e.g. to your add_wrapper function.
> >>
> >> IIUC, you are saying that the object was not held correctly by
> >> std::auto_ptr?
>
> DA> What do you mean by "correctly?"
>
> I meant that the C++ object B_Wrapper was not being held by
> std::auto_ptr. Sorry about the loose terminology.
>
> DA> I was suggesting that it was held (correctly), and then the
> DA> object's ownership was (correctly?) passed off to the auto_ptr
> DA> which was used as an argument to a function. Ownership
> DA> passing when auto_ptrs are copied is a feature of auto_ptr, so
> DA> in that sense it's "correct" to pass ownership. But I can't
> DA> say whether that's the behavior you wanted.
>
> Yes, I do want the object ownership to pass to the holder.
>
> >> I can confirm that add_wrapper was indeed being called because
> >> I added a print statement after that. Boost.Python (correctly)
> >> complains if I pass the object itself
> DA> ^^^^^^^^^^^^^^^^^
> DA> you have to be more specific; there are lots of objects flying
> DA> around. The C++ object? The Python object? From C++ or from
> DA> Python? A code example helps.
>
> Well, if I did something like this:
>
>>>> x = h.get(0)
>>>> h.add(x)
>
> Boost.Python does not allow it. Yes, I know this is a horrible thing
> to do but I just tested to see if the wrapper would accept it, it did
> not. This is what I meant by "... I pass the object". However it
> seems irrelevant now that you have clarified several things, I'll try
> and give you a minimal test case.
>
> [ PR's understanding of the ownership issues ]
>
> >> h.add calls the defined add_wrapper which expects a
> >> std::auto_ptr<B>. Since I've declared std::auto_ptr<B_Wrapper>
> >> to be implicitly convertible to std::auto_ptr<B>, when 'b' is
> >> passed to the add_wrapper function, the add_wrapper gets the
> >> pointer to the B_Wrapper, and releases b's auto_ptr that holds
> >> B_Wrapper. So the b object will not be useable anymore.
>
> DA> Good so far.
>
> Great! I got atleast something right! :)
>
> >> However h.get gets me the pointer that is stored in h:
> >>
> >>>>> x = h.get(0) x.f()
> >> B::f
>
> DA> Well, it gets you a Python 'A' object which "holds" its C++
> DA> object with a raw pointer (i.e. no ownership).
>
> Yes, h::get() (i.e. the C++ code) returns a pointer to A. IIUC, there
> is no ownership for the Python 'A' object because of the return value
> policy that I choose.
Right.
> So if I choose return_internal _reference<1>() as the return value
> policy it ties the lifetime of the returned Python A object to the
> lifetime of the Holder, h, and if h is deleted ('del h') the
> reference to the C++ object is still 'safe' (in that it won't crash
> the interpreter) to use since its a weak reference.
Right, at least until some C++ code invalidates that pointer in some
other way. You might look at the indexing suite stuff Joel has
checked in for ways to make this sort of thing even safer.
> >> Which clearly appears to work confirming what I was saying
> >> above. However I thought that func(x) should also work, but it
> >> does not and produced the error I showed earlier.
>
> DA> It looks as though that should work, to me. But I can't see
> DA> the wrapping code.
>
> [snip]
>
> DA> This is not minimal. I want to see the C++ wrapping code.
> DA> Believe it or not I have never even run Pyste.
>
> Ahh, OK. Thanks for letting me know. Next time I'll also attach the
> C++ code.
>
> As an aside, I think Nicodemus and I would be happy if you did run
> Pyste sometime. :)
Yes, so much to do; so little time. I'm interested, but I don't see
that there would be huge benefits to the community if I did that.
> Anyway, here is hopefully a minimal test case for "Problem 1".
>
<snip>
>
> $ python test_bug.py
> In add_wrapper
> Segmentation fault
Hmm, for me, in full Python debug mode with most compilers, I get
[3821 refs]
In add_wrapper
B::f
In add_wrapper
In add_wrapper
In add_wrapper
In add_wrapper
In add_wrapper
In add_wrapper
In add_wrapper
In add_wrapper
In add_wrapper
B::f
B::f
B::f
B::f
B::f
B::f
B::f
B::f
B::f
B::f
Hmm, but with gcc-2.95.3, I get:
In add_wrapper
Traceback (most recent call last):
File "foo.py", line 12, in ?
func(x)
AttributeError: 'wrapper_descriptor' object has no attribute 'f'
[5464 refs]
Hmm, the latter are using Python 2.3; the former using 2.2.2. Which
version of Python are you using?
Ouch; when I upgrade the other compilers to use Python 2.3, I get
AttributeError: 'WeakRef' object has no attribute 'f'
<spends a little bit of time with debugger>
Ick. Here's the problem: the "self" pointer is dangling by the time
B_Wrapper::f is invoked. This just reinforces my previous conclusion
that we need a more formalized way to handle these Wrapper classes,
e.g. with a common base class something like this one:
namespace boost { namespace python {
class VirtualDispatcher
{
VirtualDispatcher(PyObject* self)
: m_detached(false), m_self(self) {}
~VirtualDispatcher() { if (m_detached) Py_DECREF(m_self); }
void detach() { Py_INCREF(m_self); m_detached = true; }
object self() const
{
return object(handle<>(borrowed(m_self)));
}
private:
bool m_detached;
PyObject* m_self;
};
}}
And then we need an adopt policy something like what Daniel Wallin
has in luabind, but which is smart enough to detach() all
VirtualDispatcher objects.
> IIUC, x is a Python object that holds a pointer to the A* inside h.
> This A* actually points to the B_Wrapper object that was created by 'b
> = B()'. This has a valid default_f so should not func(x) work? Also
> note that 'x.f()' works perfectly fine.
Sure; that looks up the f attribute and calls it, while func(x) goes
through a few more layers.
> Its the func(x) that seems to fail.
> DA> struct Foo_Wrapper : Foo {
> DA> Foo_Wrapper(PyObject* self)
> DA> : m_self(self) {}
>
> DA> Foo* f() {
> DA> object self(handle<>(borrowed(m_self)));
> DA> return extract<Foo*>(self.attr("f")());
> DA> }
> DA> PyObject* m_self;
> DA> };
>
> Yes, that is possible but I'm using Pyste. Maybe we need another hook
> to insert code there?
I guess that's up to Nicodemus.
> Is there an easier way that Boost.Python could handle this better?
Easier for you; harder for me. I don't have time to solve these
problems quickly at a deep level in Boost.Python, at least, not for
free ;-)
--
Dave Abrahams
Boost Consulting
www.boost-consulting.com
More information about the Cplusplus-sig
mailing list