[C++-sig] optimizing away calls to the python runtime -- was [detecting if a function is written in python or C++]

[Mathieu Lacage]

On Thu, 2006-01-12 at 21:51 -0500, David Abrahams wrote:
> > Okay, so, to wrap this, I wrapped my Simulator class and my EventWrap
> > class:
> > struct EventWrap : yans::Event
> > {
> >         EventWrap( PyObject* self_);
> >         virtual ~EventWrap();
> >         virtual void notify (void);
> > private:
> >         PyObject *m_self;
> > };
> > EventWrap::EventWrap (PyObject* self_)
> >         : m_self(self_)
> > {
> >         Py_INCREF(m_self);
> > }
> > EventWrap::~EventWrap()
> > {
> >         Py_DECREF(m_self);
> > }
> 
> This causes a double deletion.  Also, the Py_INCREF is somewhat evil

I fail to see how it would cause a double deletion. It might create
leaks in certain conditions if the c++ code does not call delete at some
point but python should be forbidden from destroying this object because
I hold a refcount to it.

> for other reasons.  In the Boost.Python object model, C++ objects are
> owned by their Python wrappers, not vice-versa.  I think you'd be a

Yes, but my c++ object needs to ensure that the python object is live
when it calls back into it. If the Py_INCREF is not present, and if no
python variable holds a reference to the original python object, boom
when notify is called.

> lot better off if you stopped trafficking in raw pointers and used
> boost::shared_ptr<Event> instead.

Using shared_ptr would require my changing the c++ code to deal with
shared_ptr rather than raw pointers, am I right ?

> 
> > void
> > EventWrap::notify (void)
> > {
> >         call_method<void>(m_self, "notify");
> 
> If this is related to your subject line, you should know that this
> sort of old-style polymorphism does incur needless calls into Python.
> Use derivation from boost::python::wrapper<Event> instead, with
> get_override().

Ah. This is good to know. I used this call_method way because I already
had the private PyObject *m_self pointer because of the Py_INCREF so I
thought I could avoid duplicating PyObject * everywhere (in the wrapper
and in my code).

However, I don't think the boost::python::wrapper way of dealing with
this would completely eliminate the need to call back at least a bit
into python, am I wrong ?

> >         delete this;
> 
> There's that double-deletion again.


I do not think so. The C++ code is designed such that the Simulator::run
class does not deal with the memory management of the Event objects it
calls into. That is, it merely assumes that the lifetime of the Event
object is long enough to invoke Event::notify once and forget about it
after. It is up to the Event::notify and event constructor to deal with
the event's memory management. The c++ template make_event I use to
generate event forwarding classes does this automatically. It could be
argued that this is not very nice and I could even agree but, well.

As such, the wrapper I presented in this email does something like this:
  - python creates Event object which creates a c++ Event object
  - python passes python Event object to simulator.insert_in_s which
calls Simulator::insert_in_s which assumes ownership of the c++ Event *
passed.
  - python calls simulator.run which calls into Simulator::run
  - Simulator::run invokes c++ Event::notify which invokes back
Event.notify into python.
  - Event::notify deletes c++ object and should delete associated python
object.

I fail to see how the end-user API I want to offer could be implemented
in another way without changing the wrapped c++ code.

Changing the c++ code would require one of:
  - as suggested by you, using shared_ptr in the c++ code
  - Another solution might be to make the memory management issue
explicit at the C++ API level through some other mean than shared_ptr.
Adding a pair of ref/unref methods on the Event c++ class for example...

> > which allows me to write the following python code:
> > class __MyEvent__(Event):
> 
> Bad idea to name your symbols that way; __XXX__ symbols are by
> convention reserved to the Python language.

ok.

> 
> >     def set_callback (self, function, *args):
> >         self.__function_ = function;
> >         self.__args_ = args;
> >     def notify (self):
> >         self.__function_ (*self.__args_);
> >
> > def make_event(function, *args):
> >     ev = __MyEvent__ ();
> >     ev.set_callback (function, *args);
> >     return ev
> >
> > def my_nothing0 ():
> >     print "my model now=%f" % simulator.now_s ();
> > simulator.insert_in_s (4.0, make_event (my_nothing1, 99));
> > simulator.run ()
> >
> >
> > okay, so all of this stuff works really nicely but there is a catch. I
> > would like to be able to write code such as:
> >
> > simulator.insert_in_s (4.0, make_event (simulator.now_s));
> > simulator.run ()
> >
> > and ensure that once simulator.run has reached the Simulator::run c++
> > method, no python code will be executed ever. Namely, I want call to
> > make_event to generate a nice C++ Event class which will not call back
> > into python's __MyEvent__.notify before calling simulator.now_s and
> > Simulator::now_s. i.e., it should call back Simulator::now_s directly.
> > 
> > The more I think about this, the less I see a way to do this with boost.
> > Have I missed something ?
> 
> Use new-style polymorphism.

"new style polymorphism" is the use of get_override ? If so, you mean
that it can be made to call into c++ if python does not override the
function, right ?

Something like this ?

struct ACallback :  A, wrapper<A>
{
    char const* f()
    {
        if (override f = this->get_override("f"))
            return f();
        return A::f();
    }

    char const* default_f() { return this->A::f(); }
};

I don't think this is what I want to do. I would like to make python's
make_event function create the right underlying c++ Event class which,
rather than trying to call back into python does not bother with it and
just calls into c++ directly.

In case my description of my use-case is not very clear, here is the
real code for the c++ simulator:

http://spoutnik.inria.fr/code/yans/?cmd=file;filenode=7912699843995beacf89cbb81ed1533c06bb2123;file=simulator/simulator.h
http://spoutnik.inria.fr/code/yans/?cmd=file;filenode=bbc4194170ea5e790d9e296ff7037fda7954d152;file=simulator/event.h
http://spoutnik.inria.fr/code/yans/?cmd=file;filenode=9f87c99c3c90b8cb9c60a64fdb27eb60f9807385;file=simulator/event.tcc

and the documentation for the API with sample c++ code:
http://spoutnik.inria.fr/code/yans/?cmd=file;file=doc/simulator.html;filenode=303f820404c45cc68d42d38728742b030ac50176;style=raw

the python binding code is there:
http://spoutnik.inria.fr/code/yans/?cmd=file;filenode=a30e335d29d1c0fd68c21088474146c235e04303;file=python/event-wrap.h
http://spoutnik.inria.fr/code/yans/?cmd=file;filenode=fc288ba73cfb6ab80e9d1728b3e2588821690400;file=python/export-event.cc
http://spoutnik.inria.fr/code/yans/?cmd=file;filenode=231533788691a2a37d5f709ead483a5c0fd8a1c0;file=python/export-simulator.cc

and the sample python code:
http://spoutnik.inria.fr/code/yans/?cmd=file;filenode=46ff2d294159555a0178bac34169218691131864;file=python/test-simulator.py

you should be able to download and build that code if you use an x86
linux box with gcc 4.x
http://spoutnik.inria.fr/yans/releases/yans-0.3.tar.gz

It is hard for me to explain more clearly what I want to achieve because
I don't really understand enough template magic to understand how boost
python works and reformulate my problem in better terms.

thanks for your answer,
Mathieu
--