[Numpy-discussion] Getting C-function pointers from Python to C

[Dag Sverre Seljebotn]

On 04/12/2012 07:24 PM, Nathaniel Smith wrote:
> On Wed, Apr 11, 2012 at 10:23 PM, Travis Oliphant<teoliphant at gmail.com>  wrote:
>>>> In the mean-time, I think we could do as Robert essentially suggested and just use Capsule Objects around an agreed-upon simple C-structure:
>>>>
>>>>       int   id   /* Some number that can be used as a "type-check" */
>>>>       void *func;
>>>>       char *string;
>>>>
>>>> We can then just create some nice functions to go to and from this form in NumPy ctypeslib and then use this while the Python PEP gets written and adopted.
>>>
>>> What is not clear to me is how one get from the Python callable to the
>>> capsule.
>>
>> This varies substantially based on the tool.   Numba would do it's work and create the capsule object using it's approach.   Cython would use a different approach.
>>
>> I would also propose to have in NumPy some basic functions that go back-and forth between this representation, ctypes, and any other useful representations that might emerge.
>>
>>>
>>> Or do you simply intend to pass a non-callable capsule as an argument in
>>> place of the callback?
>>
>> I had simply intended to allow a non-callable capsule argument to be passed in instead of another call-back to any SciPy or NumPy function that can take a raw C-function pointer.
>
> If the cython folks are worried about type-checking overhead, then
> PyCapsule seems sub-optimal, because it's unnecessarily complicated to
> determine what sort of PyCapsule you have, and then extract the actual
> C struct. (At a minimum, it requires two calls to non-inlineable
> functions, plus an unnecessary pointer indirection.)

I think this discussion is moot -- the way I reverse-engineer Travis is 
that there's no time for a cross-project discussion about this now. 
That's not too bad, Cython will go its own way (eventually), and perhaps 
we can merge in the future...

But for the entertainment value:

In my CEP [1] I descripe two access mechanisms, one slow (for which I 
think capsules is fine), and a faster one.

Obviously, only the slow mechanism will be implemented first.

So the only things I'd like changed in how Travis' want to do this is

a) Storing the signature string data in the struct, rather than as a char*;

     void *func
     char string[1]; // variable-size-allocated and null-terminated

b) Allow for multiple signatures in the same capsule, i.e. "dd->d", 
"ff->f", in the same capsule.

> A tiny little custom class in a tiny little library that everyone can
> share might be better? (Bonus: a custom class could define a __call__
> method that used ctypes to call the function directly, for interactive
> convenience/testing/etc.)

Having NumPy and Cython depend on a common library, and getting that to 
work for users, seems rather utopic to me. And if I propose that Cython 
have a hard dependency of NumPy for a feature as basic as calli.ng a 
callback object then certain people will be very angry.

Anyway, in my CEP I went to great pains to avoid having to do this, with 
a global registration mechanism for multiple such types.

Regarding your idea for the __call__, that's the exact opposite of what 
I'm doing in the CEP. I'm pretty sure that what I described is what we 
want for Cython; we will never tell our users to pass capsules around. 
What I want is this:

@numba
def f(x): return 2 * x

@cython.inline
def g(x): return 3 * x

print f(3)
print g(3)
print scipy.integrate.quad(f, 0.2, 3) # fast!
print scipy.integrate.quad(g, 0.2, 3) # fast!

# If you really want a capsule:
print f.__nativecall__

Dag

[1] http://wiki.cython.org/enhancements/cep1000