[C++-sig] Passing memory allocated in C++ to Python

[Tony Cappellini]

> In your something class, the data type returned from allocate_memory()
> needs to be something that Python understands. Since that allocation
> function (member) will be allocating 100s of MB of memory, how will
> this memory map to a Python data type?

>>Does it have to be a Python (native) data type ? Could you explain your
>>use-case a little more ?

It needs to be a data type that can be allocated in C++ but accessible in
Python

>>Could you explain your use-case a little more ?
I have C++ code which is interfacing to a driver, from Python.
The user invokes a python function which calls my C++ code (via Boost). The
C++ code makes a C++ structure, allocates memory for the ioctl call,
puts a pointer to the allocated memory into the C++ structure (as well as
filling in other structure members), calls the ioctl and returns the
success/failure
results to Python. The user needs a way to get access to the allocated
memory, which was filled in by the ioctl call.

>>f you really need to expose the memory as a buffer to Python, you may want
to use one of the newer C APIs such as
I'm sure there are many ways to do this, but I've already chosen boost for
this project. Boost exists in order to
pass data back and forth between C++ & Python, there must be some way to do
this

The boost documentation is not very helpful, but I know many people have
shared data between Python & C++ for
a long time this

Thanks

On Tue, Jan 12, 2016 at 10:14 AM, Stefan Seefeld <stefan at seefeld.name>
wrote:

> On 12.01.2016 12:52, Tony Cappellini wrote:
> >
> > Stefan,
> >
> >
> > To: cplusplus-sig at python.org <mailto:cplusplus-sig at python.org>
> > Subject: Re: [C++-sig] Passing memory allocated in C++ to Python
> > Message-ID: <56945036.8080302 at seefeld.name
> > <mailto:56945036.8080302 at seefeld.name>>
> > Content-Type: text/plain; charset=windows-1252
> >
> > > Essentially- Python needs access to the return value from a malloc()
> call.
> >
> > >>Really ? Why ? You say your C++ code needs to access the pointer(s).
> > >>Python has no notion of "pointer", but it will happily pass around
> > >>atever data you expose from C++.
> >
> > Sorry about that Stefan, my post should have been clearer.
> >
> > I know Python doesn't know about pointers.
> > I should have said "Python needs access to the memory pointed to by
> > the memory allocated with malloc()".
> >
> >
> > >>f you expose the above class together with the constructor and thei
> > >>'call_ioctl()' member function I think you have all you need.
> >
> > In your something class, the data type returned from allocate_memory()
> > needs to be something that Python understands. Since that allocation
> > function (member) will be allocating 100s of MB of memory, how will
> > this memory map to a Python data type?
>
> Does it have to be a Python (native) data type ? Could you explain your
> use-case a little more ?
>
> >
> > Is a bytearray better to use than a list (as far as performance is
> > concerned)? Is a list better to use than a string (as far as
> > performance is concerned). Are there other data types that should be
> > considered?
>
> It all depends on what you intend to do with the memory.
>
> >
> > The user will need to access that data as bytes, words, longs, and
> > possibly sequences.
> >
> > I've tried using boost's extract, to convert the char * returned from
> > malloc() into a Python string,
> > and a Python list, but these result in compile errors.
>
> That's because the extract<>() logic is meant to be used for the
> inverse: to get access to C/C++ objects embedded into a Python object.
> Once you reflect your C++ type "something" to Python, boost.python will
> implicitly convert between the Python (wrapper) type and the C++
> "something" type for all function calls.
> However, sometimes you may still have a Python object, knowing that it
> actually contains a "something". extract<> is meant to give access to
> that (either by-value or by-reference, depending on your usage).
>
> If you really need to expose the memory as a buffer to Python, you may
> want to use one of the newer C APIs such as
> |PyMemoryView_FromMemory
> (https://docs.python.org/3/c-api/memoryview.html). Support for that
> isn't built into boost.python yet, so you'll have to add some glue code
> (e.g.
>
> http://stackoverflow.com/questions/23064407/expose-c-buffer-as-python-3-bytes
> ).
> This may actually be worth including into the library. We just haven't
> had anyone asking for such a feature - yet.
> |
> (I have used similar approaches in the past, where memory from a C++
> library I wrote is bound to a NumPy array, so the data can be accessed
> copy-free from other Python modules via the NumPy interface.)
>
> Regards,
>         Stefan
>
> --
>
>       ...ich hab' noch einen Koffer in Berlin...
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20160112/84117a10/attachment-0001.html>