Embedding, subinterpreters and dynamic types

[Thomas Jollans]

On 2010-08-13 13:22, Mikael Liljeroth wrote:
> Hi, this is my first contact with the python community so I hope this
> is the right place for my two questions.
> They are both related to embedding and extending python.
>
> I'm writing a program that dynamically loads c++ modules that extend
> the functionality of my program.
> These modules are controlled by scripts. Currently I have embedded the
> v8 javascript engine but I would also like to support scripts written
> in python. I have a special interface between the c++ modules and the
> scripts to be able to support different scripting languages. Scripts
> are separated into different 'applications' that are executed in one
> process triggered by external events (no threading, since I'm working
> in an event driven system). The scripts should execute independently
> of each other. I solved this with v8 by switching between the
> applications' contexts. However, it seems like python has some issues
> with this.
>
> 1. To do some basic context switching I assume I should use a sub
> interpreter ie create a new thread state (without any actual threads).
> I cannot find any information on how to do this except creating the
> new thread state. Is this the preferred way of doing this, if so, how
> do I switch between different thread states and execute scripts in
> them? I assume scripts running in one thread state are isolated from
> scripts running in other thread states, like global variables and such
> (except for the GIL thing?) ? I do not care much about security
> problems like file descriptor issues and extension issues right now.
>   
Py_NewInterpreter does indeed sound like what you want. I have a feeling
you're going to run into problems here sooner or later, say, when
extension  modules that aren't yours use static variables.

As for switching interpreters, after skimming the docs, I'm guessing
you're looking for PyEval_AcquireThread

http://docs.python.org/py3k/c-api/init.html#PyEval_AcquireThread

> 2. When a new C++ module is loaded into my program I want to create a
> representation of it in Python. Not as a type, but rather just return
> a single object instance reference of a new type representing the
> exported attributes and methods of the module's C++ class. In other
> words, I want to be able to do something like this obj =
> application.getObject("objectId"); where objectId is the name of an
> instance of that c++ module type. I do not want the script to be able
> to create new instances of this type, since this is done via xml in my
> program. Hence I would also like to be able to associate my C++ object
> with my returned python object to be able to manipulate it when
> methods are called on 'obj'. So how do I define a new type on the fly
> and create an instance of it, associated with a c++ pointer and return
> it in the c function representing the python method 'getObject' in my
> application module?
>   

How difficult this is obviously depends somewhat on how varied the types
you create are, but in principle, there's no difference between creating
a type statically and creating one on-the-fly: all you need to do to
create a type is create a PyTypeObject structure with suitable function
pointers. If you want to make it impossible to create a second instance
of your type, you can simply create a tp_new or tp_init that raises an
exception.

If you have a good look at
http://docs.python.org/py3k/extending/newtypes.html , you'll notice that
most of the "creating a type" involves  writing a static PyTypeObject
structure -- you'll just have to allocate that yourself, and you should
be good.