real-time monitoring of propriety system: embedding python in C or embedding C in python?

[Stefan Behnel]

Bas, 05.02.2013 16:10:
> at work, we are thinking to replace some legacy application, which is a
> home-grown scripting language for monitoring and controlling a large
> experiment. It is able to read live data from sensors, do some simple
> logic and calculations, send commands to other subsystems and finally
> generate some new signals. The way it is implemented is that it gets a
> chunk of 1 second of data (thousands of signals at sample rates from 1Hz
> to several kHz), does some simple calculations on selected signals, does
> some simple logic, sends some commands and finally computes some 1Hz
> output signals, all before the next chunk of data arrives. The purpose
> is mainly to monitor other fast processes and adjust things like process
> gains and set-points, like in a SCADA system. (I know about systems like
> Epics and Tango, but I cannot use those in the near future.) It can be
> considered soft-real time: it is desirable that the computation finishes
> within the next second most of the time, but if the deadline is missed 
> occasionally, nothing bad should happen. The current system is hard to
> maintain and is limited in capabilities (no advanced math, no
> sub-functions, ...).
> 
> I hope I don't have to convince you that python would be the perfect
> language to replace such a home-grown scripting language, especially
> since you than get all the power of tools like numpy, logging and
> interface to databases for free. Convincing my colleagues might cost
> some more effort, so I want to write a quick (and dirty?) demonstration
> project. Since all the functions I have to interface with (read and
> write of live data, sending commands, ...) are implemented in C, the
> solution will require writing both C and python.

Or Cython and Python. Cython is a Python dialect that compiles down to C,
so you get native C/C++ interfacing and speed without leaving the wonderful
world of Python or the CPython runtime. It also comes with a lot of cool
features for efficient data processing that interface efficiently with NumPy.

And, yes, you'll most likely end up using NumPy in one way or another. It's
basically Python's data integration layer for high-performance and large
data computation.


> I have to choose between two architectures:
> 
> A) Implement the main program in C. In a loop, get a chunk of data using
> direct call of C functions, convert data to python variables and call an
> embedded python interpreter that runs one iteration of the user's
> algorithm. When the script finishes, you read some variables from the
> interpreter and then call some other C-function to write the results.
> 
> B) Implement the main loop in python. At the beginning of the loop, you
> call an embedded C function to get new data (using ctypes?), make the
> result readable from python (memoryview?), do the user's calculation and
> finally call another C function to write the result.
> 
> Are there any advantages for using one method over the other? Note that
> I have more experience with python than with C.

When it comes to interfacing, there is no difference between both. The only
question is really what starts up the application. This tends to be a bit
simpler if it's Python, but otherwise, it's just a one-time boilerplate
code writing thing.

The usual way to go about it is to write an extension module (commonly in
Cython) and import that from within the running Python runtime. In that
module (a shared library) you implement a wrapper around your C-level API,
often including some more high-level functionality that simplifies and
beautifies the underlying C-ish API for you. Importing that module can be
done using the normal "import" statement from Python code if the Python
runtime is running the app, and is more commonly registered at the C level
if you start Python from C.

That being said, I'd encourage you to go for the "Python runs it all"
approach first, because it tends to be less overhead and less fiddling to
get it working.

Stefan