[C++-sig] Passing memory allocated in C++ to Python

[Stefan Seefeld]

On 12.01.2016 12:52, Tony Cappellini wrote:
>
> Stefan,
>
>
> To: cplusplus-sig at python.org <mailto:cplusplus-sig at python.org>
> Subject: Re: [C++-sig] Passing memory allocated in C++ to Python
> Message-ID: <56945036.8080302 at seefeld.name
> <mailto:56945036.8080302 at seefeld.name>>
> Content-Type: text/plain; charset=windows-1252
>
> > Essentially- Python needs access to the return value from a malloc() call.
>
> >>Really ? Why ? You say your C++ code needs to access the pointer(s).
> >>Python has no notion of "pointer", but it will happily pass around
> >>atever data you expose from C++.
>
> Sorry about that Stefan, my post should have been clearer.
>
> I know Python doesn't know about pointers.
> I should have said "Python needs access to the memory pointed to by
> the memory allocated with malloc()".
>
>
> >>f you expose the above class together with the constructor and thei
> >>'call_ioctl()' member function I think you have all you need.
>
> In your something class, the data type returned from allocate_memory()
> needs to be something that Python understands. Since that allocation
> function (member) will be allocating 100s of MB of memory, how will
> this memory map to a Python data type?

Does it have to be a Python (native) data type ? Could you explain your
use-case a little more ?

>
> Is a bytearray better to use than a list (as far as performance is
> concerned)? Is a list better to use than a string (as far as
> performance is concerned). Are there other data types that should be
> considered?

It all depends on what you intend to do with the memory.

>
> The user will need to access that data as bytes, words, longs, and
> possibly sequences.
>
> I've tried using boost's extract, to convert the char * returned from
> malloc() into a Python string,
> and a Python list, but these result in compile errors.

That's because the extract<>() logic is meant to be used for the
inverse: to get access to C/C++ objects embedded into a Python object.
Once you reflect your C++ type "something" to Python, boost.python will
implicitly convert between the Python (wrapper) type and the C++
"something" type for all function calls.
However, sometimes you may still have a Python object, knowing that it
actually contains a "something". extract<> is meant to give access to
that (either by-value or by-reference, depending on your usage).

If you really need to expose the memory as a buffer to Python, you may
want to use one of the newer C APIs such as
|PyMemoryView_FromMemory
(https://docs.python.org/3/c-api/memoryview.html). Support for that
isn't built into boost.python yet, so you'll have to add some glue code
(e.g.
http://stackoverflow.com/questions/23064407/expose-c-buffer-as-python-3-bytes).
This may actually be worth including into the library. We just haven't
had anyone asking for such a feature - yet.
|
(I have used similar approaches in the past, where memory from a C++
library I wrote is bound to a NumPy array, so the data can be accessed
copy-free from other Python modules via the NumPy interface.)

Regards,
        Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...