[Numpy-discussion] Numpy array from ctypes pointer object?

[Mark Heslep]

Albert Strasheim wrote:
> Hello all
>
> Various people wrote:
>   
>>>  Im curious though:  the several projects recently using ctypes
>>> and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come
>>> across the issue of using a creating a numpy array from a ctypes
>>> pointer.  Ill have to look further.
>>>
>>>       
>> It depends on whether or not the library creates memory or not.  Not
>> every library manages memory (some expect you to pass in memory already
>> owned --- this is easy to do already with ctypes and numpy).
>>     
>
> I see two main models for sending back data from C land.
>
> One is to have the C function allocate memory and return the pointer to the
> caller. This raises the question of who owns the memory. It also leads to
> some *very* interesting crashes on Windows when you start freeing memory
> allocated in code linked against one runtime in code linked against another
> runtime. I think, if you have the option, avoid this kind of thing at all
> costs.
>
> The other model has functions that take their normal arguments, and pointers
> to buffers where they can store their results. Typically the caller would
> also specify the size of the buffer. If the buffer is large enough for the
> function to do its work, it uses it. If not, it returns an error code. This
> way, one entity is always in charge of the memory.
>
> If you're writing C code that you plan to use with ctypes and NumPy, I think
> the second model will lead to more robust code and less time spent debugging
> crashes and memory leaks.
>   
Yes, I agree, C libs written with your model #2 would make life much 
easier to create robust wrappers. Many of the ones that come to my mind, 
and deal with array like data and thus relevant, are model #1 types. I 
think the reason this is so goes something like this:

Joe Foo developer & friends writes an entire library API in C. To make 
the API complete and easy to use Joe includes getting started 'make the 
data' C functions. Examples:
SDL: SDL_surface* SDL_CreateRGBSurface(params)
Opencv: IplImage* cvCreateImage(params), ...
libdc1394: uint_t * capture buffer //created by the OS 1394 driver
etc.

If the Joe doesn't do this then the first thing Joe's 'users' must do to 
create the simplest application is call malloc, sizeof( figure out what 
element type...), blah, blah, i.e. do lots of memory management having 
zilch to do with the problem at hand. Further, Joe, being conscientious, 
includes boat loads of examples all using the 'make the data' calls.

> libsvm (which I'm wrapping with ctypes) mostly follows the second model,
> except when training new models, in which case it returns a pointer to a
> newly allocated structure. To deal with this, I keep a pointer to this
> allocated memory in a Python object that has the following function:
>
> def __del__(self):
>     libsvm.svm_destroy_model(self.model)
>
>   
Nice
> By providing this destroy function, libsvm avoids the problem of mixing
> allocation and deallocation across runtimes.
>
> Regards,
>
> Albert
>
>   

-Mark