[Numpy-discussion] Ctypes support in NumPy

[Travis Oliphant]

Albert Strasheim wrote:
> I did a few tests and this seems to work nicely:
>   

Hey Albert,  I read the post you linked to on the ctypes mailing list.  
I hope I didn't step on any toes with what I did in NumPy.   I was just 
working on a ctypes interface and realized that a lot of the cruft to 
convert to what ctypes was expecting could and should be handled in a 
default way.  The conversion of the shapes and strides information to 
the "right-kind" of ctypes integer plus the inclusion of ctypes in 
Python 2.5 was enough to convince me to put some kind of hook into the 
array object.  I decided to make the ctypes attribute return an object 
so that the object could grow in the future additional attributes and/or 
methods to make it easier to interface with ctypes.

I looked a bit at the source code and was disappointed to see that the 
_as_parameter_ approach is pretty limited.  While there is talk of 
supporting a tuple return of _as_parameter_ in the source code comments, 
there is no evidence in the source itself of supporting it.

There is also the changed way of handling additional arguments when 
argtypes is set on the function which uses the from_param method.  
Unfortunately, as Thomas responds to your post, the from_param method 
must be on one of the ctypes to work.  You have to add support 
specifically for one of the c-data types.  

I think the _as_parameter_ approach returning a tuple that could be 
interpreted as the right ctype was better because it let other objects 
play the ctypes game. 

Basically, what you need is a type-map just like swig uses.  But, now 
that ctypes is in Python, it will be slower to change.  That's a bit 
unfortunate. 

But, ultimately, it works fine now.  I don't know what is really gained 
by applying an argtypes to a function call anyway --- some kind of 
"type-checking".  Is that supposed to be safer. 

For NumPy extension modules, type checking is only a small part of the 
memory-violation danger.  In-correct array bounds and/or striding is far 
more common - not-to mention unaligned memory areas and/or unwriteable 
ones (like a read-only memory-mapped file). 

Thus, you're going to have to write a small "error-checking" code in 
Python anyway that calls out to the C-library with the right 
arguments.   So, basically you write an extension module that calls 
c-code just as you did before, but now the entire "extension" module can 
all be in Python because the call to an arbitrary C-library is made 
using ctypes.

For arrays, you will typically need to pass one or more of  the data, 
the dimension information, the stride information, and the number of 
dimensions.  The data-type will be known about because function calls 
usually handle only a specific data-type.  Thus, I started with a ctypes 
object that produces this needed data in the format that ctypes needs, 
so it can be very easy to use an array with the ctypes module.

Frankly, I'm quite impressed with the ease of accessing C-code available 
using c-types.   It quite rivals f2py in enjoyment using it.

One thing I like about c-types over Pyrex, for example is that it lets 
you separate the C-code from the Python code instead of "mixing it all 
together"

I wouldn't be surprised if c-types doesn't become the dominant way to 
interface C/C++ and possibly even Fortran code (but it has tougher 
competition in f2py) once it grows up a little with additional ease-of-use.


-Travis