Embedding Python

[Tom Cocagne]

 I've recently been working with both extending and embedding Python and
have been wondering about some similar issues. Something that just recently
occurred to me is that you might be able to use Python's Global Interpreter
Lock to implement a fairly simple multi-threaded lock-step solution.
Consider the following:

1. Your program starts up, Initializes the Python interpreter, and acquires
the GIL (Global Interpreter Lock).

2. The program then creates a new thread to run the Python scripts. In
theory, this thread will immediately block in an attempt to acquire the GIL

3. In your main GUI loop, you set a periodic timer that expires at a rate of
say.... 20 Hz (this would likely need tuning) that, upon expiry, uses the
following pseudo-code:
 global_interpreter_lock.unlock()
 # Give Python approximately 20 msec of execution time
 main_application_thread.sleep(20 milliseconds)
 global_interpreter_lock.lock()
 .... continue normal QT processing .....

I haven't put a whole lot of thought into it yet but on the surface it seems
like an approach along these lines would be fairly easy to implement and
should guarantee that only 1 thread is ever accessing your application.

Using this approach would involve at least three tuneable parameters. The
GUI timer to indicate how often Python should be given some CPU time, the
delay to determine how long the Python interpreter should be allowed to
run, and byte-code check interval Python uses to release the GIL (described
in the C-API documentation on Python+threads). Getting all these right
might take some time.

  Again, I have never actually tried this so there could be some demons
lurking in the shadows. Anyone see problems with this approach?

 Cheers,

 Tom


Martin wrote:

> Greetings,
> 
> I am new to python and wish to embed python in an 3D graphics
> application to provide application automation. The high level goal is to
> be able to drive my app from a script for batch job like behavior rather
> than via the GUI (ie. I would like to run a script and see those changes
> reflected in the GUI as if the user had clicked buttons, etc.) The
> application is written in C++ and uses QT for the GUI.
> 
> I have read the python docs on python.org and (using SWIG) have been
> able to create Python shadow wrappers around the C++ classes I want to
> allow users to have been able to call them via python scripts. This part
> works well. However in terms of embedding, I feel like I am missing
> something.
> 
> For those of you familiar with QT, you know that QT apps run in a
> single-thread (essentially). Although QT supports threads, a
> QApplication based program is (by default) single-threaded. Callbacks
> (slots in QT speak) can be hooked up based on a users actions, but these
> callbacks all run in the main program thread.
> 
> So here is my question ... What's the best way to integrate Python into
> an app such as the one I am describing.
> 
> Here's what I have though of ...
> 
> Alternative #1
> Add boiler plate python embedding code (ie. Py_Initialize,
> PyRun_SimpleFile(...), Py_Finalize) in a QT callback. Unfortunately,
> since the QT app is single threaded, this will wedge up the GUI, until
> the script is done. Not good.
> 
> Alternative #2
> Keep the app as a single threaded app and interpret the script piece
> meal (ie. one statement at a time) at some timed interval. Allow for GUI
> updating between statement interpreting. Unfortunately, groveling over
> the Python C API, this does not look particularly easy either and would
> more or less require cutting/pasting much of the interpreter guts in my
> app. Furthermore, incomplete statements (ie. lines ending in : ) seem to
> present a problem to PyRun_SimpleString.
> 
> Alternative #3
> Split the app up so that it is broken up into 2 threads (GUI and guts).
> The interpreter would run in the gut thread and change the internal data
> store. The GUI would then have to be smart enough to update itselft.
> Unfortunately, this requires major app re-architecting (something I am
> trying to avoid).
> 
> Alternative #4
> Spawn an interpreter thread using multi-threaded interpreter calls. This
> would then require locking/unlocking shared resources between
> Interpreter thread and the rest of the app. It feels like if I am going
> to do this #3 may be a cleaner approach.
> 
> I am sure someone out there has done this before.
> 
> For me, the key point is that I want to add this functionality without
> disrupting the entire product. I am trying to evaluate what is
> architecturally the best approach and what will give me the most bang
> for my buck.
> 
> Any help would be appreciated.
> 
> Cheers and Thanks,
> 
> Martin