[Numpy-discussion] Numpy array from ctypes pointer object?

[Travis Oliphant]

Mark Heslep wrote:
> Travis Oliphant wrote:
>   
>> The problem here is that from Python NumPy has no way to create an 
>> ndarray from a pointer.   Doing this creates a situtation where it is 
>> unclear who owns the memory.  It is probably best to wrap the pointer 
>> into some kind of object exposing the buffer protocol and then pass 
>> that to frombuffer (or ndarray.__new__).
>>     
> Yep thats where I just ended up:
>
> from ctypes import *
> import numpy as N
> ...
> func = pythonapi.PyBuffer_FromMemory
> func.restype = py_object
> buffer = func( im.imageData, size_of_the_data )                 
> <----imageData = ctypes.LP_c_ubyte object
> return N.frombuffer( buffer, N.uint8 )
>
> Works!  

Nice job!

>  Im curious though:  the several projects recently using ctypes 
> and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come 
> across the issue of using a creating a numpy array from a ctypes 
> pointer.  Ill have to look further.
>
>   
It depends on whether or not the library creates memory or not.  Not 
every library manages memory (some expect you to pass in memory already 
owned --- this is easy to do already with ctypes and numpy).

>> When an ndarray is using memory that is not its own, it expects 
>> another object to be "in charge" of that memory and the ndarray will  
>> point its base attribute to it and increment its reference count.   
>> What should the object that is "in charge" of the memory be?
>> Perhaps a suitable utility function could be created that can work 
>> with ctypes to create ndarrays from ctypes memory locations and either 
>> own or disown the data.
>>
>>     
> I suppose that is still the case w/ PyBuffer_From.. above. That is, the 
> underlying im.imageData pointer can not be released before buffer.
>   

Yes,  you are right.  It is the memory that is most critical.  Who owns 
the memory pointed to by im.imageData?  When will it be freed?   The 
ndarray object is holding a reference to the Python buffer object which 
is just *hoping* that the memory it was initiated with is not going to 
be freed before it gets deallocated (which won't happen until at least 
the ndarray object is deallocated).

So, managing the memory can be a bit tricky.   But, if you are sure that 
im.imageData memory will not be freed then you are O.K.

-Travis