[C++-sig] Re: [newbie] Question on embedding/extending with boost::python

[Brett Calcott]

"Dirk Gerrits" <dirk at gerrits.homeip.net> wrote:
>
> Chris Heald wrote:
> > Hi all --
> >
> > I'm working on an app that is going to be a C++ code engine which
> > will be heavily extended through a series of Python scripts
> > (objects, really). My desired requirements are as follows:
> [snip]
> > having trouble is with c and d - how do I give a Python app hooks
> > into an existing C++ state machine? I know I can write an extension
> > easily enough, but by my understanding, that will simply give the
> > Python object access to a series of functions, and not the objects
> > contained in the state machine that the interpreter is running
> > under. I could use "pull" operations, wherein the C++ app polls the
> > objects for changed state in a loop and updates its sate
> > accordingly, but that seems that it wouldn't perform nearly so well
> > as if the Python scripts were able to update the C++ state machine
> > themselves, so I'm avoiding that route.
>
> I'm not entirely sure what your problem is.
>
> Have you looked at libs/python/test/embedding.cpp? It shows how your
> embedded interpreter can get access to the internals of the running
> C++ program. (Specifically, its classes.) Perhaps you can use that as
> a starting point?
>
> > I had a co-worker suggest that I use Python as my primary execution
> > environment instead of C++. While this would admittedly make some
> > things easier, how much performance would I sacrifice with this
> > route? I'm building a 3D application, and it doesn't seem that
> > Python would be well-suited to such an application in terms of
> > performance. There is potential for literally tens of thousands of
> > instances of these objects to exist in this state machine, and I
> > certainly don't want to hinder performance by moving the primary
> > execution into an interpreted environment rather than a compiled
> > environment.
>
> Well, my common sense tells me that you may very well be right. But
> the important point is that there's really no way to tell, except for
> trying it out. (Profiling is the key.) And surely, writing the whole
> thing in Python (except for the critical sections) takes less time
> then writing the whole thing in C++ with just some things written in
> Python?
>
> Taking this road could very well lead to disappointing performance as
> you say. But then you'd still have gained experience from making the
> Python prototype, and its C++ bits can most-likely be recycled anyway.
>
> I don't really feel qualified to advise you to do one thing or the
> other. Embedding is sometimes the right/efficient way to do things,
> but such situations are becoming more and more rare. I am guessing
> that yours is such a situation, but you could argue that making a
> decision on such a guess is premature opimization (and therefore evil
> ;)). Perhaps others here can provide additional insights.
>

I'm just about to head down a similar path. I have written some fairly
intensive multi-agent simulations in C++, and I am now moving my primary
development to python -- with a core engine being written in C++.

I had previously embedded python in my c++ app so that I could script my
agents, but the result was very slow as this effectively did this
(vastly simplified)

while (engine_running)
{
    for (agents_container:iterator a = agents.begin(); a !=
        agents.end(); ++a)
    {
        // here we call out to python for each agent
        (*a).update_yourself_via_python()
        update_visual_stuff()
    }
}

The calls to python are in the inner loop and this is slow.

What I am doing now is different - I am constructng an engine in C++,
making all the agents and world parameters accessible to python, BUT
the update main loop remains entirely in C++:

import engine # boost python module

while engine_running:
    engine.cycle()

What do I get out of this?

It makes it really easy to do all the mundane configuration stuff in
python. Hence I can do this...

for i in range(100):
    a = engine.Agent() # a C++ object export to python
    a.size = 10
    a.speed = 4
    a.x = random.uniform(0, 100)
    a.y = random.uniform(0, 100)
    engine.add_agent(a)

...because I don't care how fast the setup takes. I can also pause the
engine and inspect my agents, and interactively change them. Again
something that may be relatively slow because of all the python calls,
but it doesn't matter because it is happening in "user time" - whilst I
am interacting with it.

I can also change this stuff without going through any recompilation.

So, the framework, interface, and configuration are all in python, but
the bit that needs to be fast - cycling through thousands of agents and
updating them for each time step - is all in c++.

Anyway, that's the plan.. it's early days yet.

So...
What bits do you want to happen in python? And where do these lie in the
your program update loop (assuming it is some interactive graphics
program). I think you need to avoid crossing the python/C++ inside any
inner loops.


HTH,
Brett