[C++-sig] Re: The "return always existing pointer throws dangling error" problem

[David Abrahams]

"Niall Douglas" <s_sourceforge at nedprod.com> writes:

> On 21 Oct 2003 at 21:37, David Abrahams wrote:
>
>> >> No, they have many relationships.  I'm not sure which kind of
>> >> relationship you have in mind, though.
>> >
>> > I'm just surprised, that's all. 
>> 
>> I haven't confirmed your thought about "zero relationship", so what is
>> there to be surprised about?  I don't know what you mean, and you
>> don't know what Boost.Python is doing AFAICT.
>
> I based that statement on what you had previously described as the 
> operatings of BPL, just there in what I replied to.
>
> That's what's so very interesting in our dialogues. 

I'm much less interested in the sociological aspects of this than you
are, but...

> You explain something because I didn't understand. I then read your
> explanation and either (a) make statements from what your
> explanation infers which turn out to be wrong or (b) apply your
> statements to what /you/ consider to be a completely different topic
> of conversation and again, I get it wrong.

...in *one* concession to your interest, and in the hope that you'll
reevaluate the way you're posting, I'll point out that I think your
analysis of is happening between us is much more complicated than it
needs to be.  A simpler analysis might yeild better results:

  You asked a question "are you saying they have absolutely zero
  relationship?", and then, regardless of the fact that I told you
  that they are very related, you told me you're surprised.
  Presumably, you're surprised to hear that "they have absolutely zero
  relationship", something I just got finished denying in terms that
  couldn't be plainer.  Unless my presumption about the reason for
  your surprise is wrong, you've simply failed to read or respond to
  what I actually wrote.

In any case, I'm done discussing our communication disconnect, and
I'm returning to technical issues for the remainder of this thread.

>> > I had imagined that in the situation of C++ calling Python calling
>> > C++, some form of metadata would need to accompany the data obtained
>> > from C++ so you know you can throw away its temporarily constructed
>> > container as the data reenters the C++ world. 
>> 
>> <sigh> Specifics and precision, please.  Guessing what you mean when
>> you say things like "the data obtained from C++" in a context where
>> data flows through the Python/C++ boundary twice in each direction is
>> interesting, but not very productive.
>
> To me at least, it was extremely clear from across the three emails I 
> have posted prior to stating that statement that "the data obtained 
> from C++" was that which the python function (invoked by call_method 
> which was invoked by a virtual function call) had obtained by calling 
> C++.
>
> To frame this in terms of the example I sent you, the python .def 
> getMetaClass() must return a FXMetaClass in order to fufil its 
> interface contract. That FXMetaClass is "the data obtained from C++" 
> because python must call C++ to get one (since it's not a wholly 
> python class). Therefore, C++ calls virtual C++ method 
> getMetaClass(), virtual C++ method getMetaClass() calls python .def 
> getMetaClass(), and it must return a FXMetaClass.
>
> I really, really hope you know what I mean now?

Close enough, I hope.  You said "C++ calling Python calling C++", but
the chain you just described starts with a Python call and ends with a
Python call.  I'm going to assume that "C++ calling Python calling
C++" begins after the word "Therefore" above and you left off the last
call in the chain, where the implementation of the Python getMetaClass
method which calls the C++ default implementation.

I've done some more thinking about this problem and I think I
understand how it should be solved.  By way of explanation, let me
show why I think the "metadata" approach isn't the best solution.  In
the example you sent me, as you surmised, the immediate problem is
that, at the there's no knowledge of the true ownership relationship
between the returned Python FXMetaClass object and the C++ object it
holds at the point where the Python result object must be converted to
C++ inside of call_method.

Even if you were to get past that problem, however, you'll have a
worse problem when you try to call its getClassName member function,
for a similar reason (**).  getClassName returns a char const*, so the
virtual function wrapper override invokes

  call_method<char const*>(...)

which invokes a Python wrapper function for the C++ default
implementation, which returns M's className member, a char const*.
This char const* must be converted into a Python string, and there's
simply no way for a Python string to hold its contents by reference:
they *will* be copied; that's out of my control.  Even if I could
attach metadata to the Python string describing its origin, it
wouldn't do any good (***).

Furthermore, the metadata approach, like the one the library currently
takes, misses an optimization opportunity, causing Boost.Python
virtual functions that *haven't* been overridden in Python to be much
slower than they could be.  IMO the best way to solve your problem is
to prevent any Python code at all from executing in the case where the
virtual function is not overridden in Python.  In other words, a C++
override which invokes call_method should first check to see if the
function's Python implementation has been overridden, and if not,
invoke the default implementation instead.

This actually has the potential to considerably simplify wrapper
class construction, as a separate default function implementation
wouldn't be needed.  It might look like:

  struct FX_FXApp_Wrapper
      : FX::FXApp, boost::python::dispatcher<FX::FXApp>
  {
      FX_FXApp_Wrapper(PyObject* self_):
          FX::FXApp(), boost::python::dispatcher<FX::FXApp>(self_) {}

      const FX::FXMetaClass* getMetaClass() const
      {
          if (method<const FX::FXMetaClass*> m = get_override("getMetaClass"))
              return m();
          else
              return FXapp::getMetaClass();
      }
  };


or even, possibly:

  struct FX_FXApp_Wrapper
      : boost::python::dispatcher<FX::FXApp>
  {
      FX_FXApp_Wrapper(PyObject* self_)
        : boost::python::dispatcher<FX::FXApp>(self_) {}

      const FX::FXMetaClass* getMetaClass() const
      {
          if (method m = get_override("getMetaClass"))
              return m();
          else
              return FXapp::getMetaClass();
      }
  };

None of this is to say that there's something wrong with the
"metadata" approach. My proposal isn't completely general - it doesn't
apply in cases where virtual function default implementations aren't
involved.  The metadata probably would be implemented with a virtual
function on Holder classes which would return true iff the lifetime of
the held C++ object was dependent on that of the Python object.

Sadly, I don't know when I'd have time to implement any of this; I'm
desperately trying not to fall behind on a book I'm writing.

(**) I'm assuming that in the interests of minimality, your example
leaves the "virtual" off of FXMetaclass::getClassName, but if I'm
wrong please just imagine that it is a virtual function for the sake
of argument -- it's certainly a common case to have virtual functions
returning char const*.

(***) yes, we might be able to build a subclass of Python's string
which contains an char const* pointing to the original characters in
addition to the copied characters, but that's a hack because the
original characters can go out-of-synch with the copied ones.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com