Best way to go about embedding python

[Nathan Ernst]

In regards to performance of Lua vs Python, I don't have enough (near zero
experience) with Lua to comment there.

But in regards to embedding in a game, the only experience I have w/ Python
being embedded is while working on modding Civilization IV. What I saw
there just made me nauseous.

The reasoning for moving to Lua in Civilization V was performance & memory
usage - but they completely screwed up exposing the C++ internals to Python
in Civ IV. They copied object instances & containers to the Python
embedded, then marshalled them back to the C++ code when the Python code
was finished. It was horribly inefficient.

I don't have much experience hosting Python inside of a C/C++ app, but I've
10 years of experience consuming C++ APIs from Python, mostly wrapped using
Boost Python. I always did my best to minimize data marshalling. The Python
objects exposed the C++ objects' interface, directly while holding a
pointer (usually shared) to underlying C++ objects and only marshalling on
method/property calls where needed (i.e. w/ strings, dates, datetimes).
Containers were wrapped to expose a Python interface to an underlying C++
container w/ no copying.

TLDR; Civilization IV shows an antipattern about how to embed Python in a
native app and expecting anything resembling good performance. Civ IV used
Boost Python, but they did it in an extremely poor, suboptimal fashion.
Boost Python itself can be very performant, and I love the library. Boost
Python, however does suffer from a learning curve that is akin to
attempting to scale a 1000ft cliff with no rope or tools or even shoes. Due
to the heavy use of template metaprogramming in C++, compiler errors if
you're even just slightly off can be extremely hard to decipher. I've not
used it lately or on modern compilers, so I don't know if it's gotten
better, but tools like pretty_make.py (can find on github) can help filter
out the compiler spewage from the real compiler error.

PS. I don't think that Python couldn't/shouldn't be used for scripting a
game, I'm just offering up Civ IV as an example of how it can be done
horribly wrong.

Regards,
Nate

On Sun, Nov 13, 2016 at 7:03 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Nov 14, 2016 at 11:40 AM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
> > On Mon, 14 Nov 2016 12:23 am, Chris Angelico wrote:
> >
> >> Python without its stdlib is a disappointingly featureless
> >> language :)
> >
> >
> > I don't think that's true. You can do a lot in Python without any
> imports in
> > your code:
> >
> > - basic string processing
> > - BigNum (int) and float arithmetic
> > - lists and tuples
> > - dicts
> > - lazy processing of iterators
> > - custom functions and classes
> > - functional style zip, map and filter
> >
> > and more.
> >
> > Some parts of the standard library are required for built-in functions,
> e.g.
> > the io module is needed to open files. But I think you could probably
> > reduce the standard library by, oh, 90% and still have a decent language
> > for game scripting.
>
> Perhaps what would be more accurate is that Python without its stdlib
> is as featureless as (say) Lua, but still without being safe to embed.
> Yes, it's technically usable, but you really don't have a lot. And
> remember, some of the stdlib is what we call builtins; if you embed
> Python and engage "restricted mode", it's by emptying out the
> builtins. You're pretty much down to three things:
>
> 1) Stuff that you could pass to ast.literal_eval
> 2) Control flow statements
> 3) Hacks that get you around the limitations, thus proving that
> Python-without-stdlib is really really hard to actually test anything
> with.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>