Efficient data flow between orthogonal C extensions?

[Magnus Lie Hetland]

I'm about to write a component-based data flow program, and I'm
pondering the design, partly when it comes to efficiency. Basically,
the system will instantiate a set of objects, set up in a directed
network-like structure, and they will pass data among themselves in a
stream-like fashion.

Performance may not be an issue to begin with, and most components
will probably be written -- or at least prototyped -- in Python.
However, if certain components perform hard computations, it should be
possible to replace them with extension objects written in C (or C++
or Fortran or whatever).

What I have described so far shouldn't be hard to achieve. However, I
have one addition requirement: If one object is passing data to
another, and both are written in C, the data passing should be as
efficient as possible. In other words, going through a data-passing
mechanism that is written in Python would be undesirable -- or so I
think. The exact format of the data may be modified according to the
mechanism chosen.

The simplest solution I've found so far is to pass chunks (arrays) of
c structs. Then I _can_ simply go through Python, and efficiency can
be had by lumping may structs together (i.e. the arrays have varying
lengths). In other words, a form of buffering. One drawback with this
is that it may require quite a bit of parameter tuning and tradeoffs
(e.g., buffer length vs. idle time).

A possibly obvious solution would be to write a C API for the
extension objects, and for the objects to interact directly. The
problem with this is that all the extensions would (as I understand
it) have to know about each other (in the form of #import statements),
and that is clearly undesirable. Each extension should only need to
know what it is told by the main coordinating system (which is written
in Python).

Are there any existing (hopefully elegant) solutions for this sort of
thing? Any ideas? I know I may be getting ahead of myself, thinking
about optimization before I've implemented anything, but I believe
this may turn out to be an important part of the basic design... :)

- M

-- 
Magnus Lie Hetland                "In this house we obey the laws of
http://hetland.org                 thermodynamics!"    Homer Simpson