[SciPy-User] Share memory between python an C++

[Gael Varoquaux]

Hi,

On Sat, Oct 27, 2012 at 02:54:06AM +0200, FDM wrote:
> I have a couple of functions in the form of shared C++ libraries, and
> want to use them from within python. Some of them involve big chunks of
> data which could be represented easily using numpy data types.
> Therefore, I am searching for a way to call the C++ function, pass a
> reference or pointer as argument, pointing to memory I have allocated
> in python, such that I can use the result of the function w/o copying.

You should rely on the numpy support in Cython, and use Cython to call
the C++ function.
See for instance the following Cython file that we use in the
sickit-learn to call the Murmurhash library:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/murmurhash.pyx
This file is somewhat lacking an example of passing an array as a pointer
to C code. This can be done by passing the '.data' attribute of the
array, that is converted by Cython to a pointer. The following file has
examples of this:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/liblinear.pyx

Be sure to read the Cython docs. They are excellent :)

Also, I have written an small example showing how to do something similar
in Cython, which is to use memory allocated in C++ without copy, and
with clean garbage collection. This is much harder, I find, and I try to
avoid it, but it comes in handy sometimes:
http://gael-varoquaux.info/blog/?p=157

Hope this helps,

Gaël