[SciPy-Dev] Enabling use of scipy.special from Numba

[Joshua Wilson]

Hey everyone,

For a while I've been thinking about how we can improve the
compatibility of `scipy.special` with Numba. There is a way to do that
currently by using `cython_special`:

https://github.com/numba/numba/issues/3086

but that has a few drawbacks:

(1) it's not very user friendly
(2) it relies on the (I think?) private implementation details of
Cython's `__pyx_capi__` (including the name mangling scheme used for
fused functions)
(3) it's clumsy for complex functions

I think (3) requires changes on the Numba side, e.g.

https://github.com/numba/numba/issues/3860

For (1) you could imagine us implementing another Numba specific API,
but that runs into some immediate issues:

- We'd need Numba as a dependency (which has been discussed:
https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html)
- We don't want to just keep implementing new APIs

For those reasons I'd prefer to find another solution. In an ideal
world I'd like to solve (2) in such a way that it's easy for a
third-party library to wrap `cython_special` in Numba jitted
functions. There is a solution to that already that doesn't involve
`__pyx_capi__`: a third-party library writes some Cython glue that
exports PyCapsules of function pointers to the specializations of all
the `cython_special` functions and then you grab those using ctypes to
construct the jitted functions. For the Cython side of that see the
docs on specialization:

http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-specializations

But that feels rather convoluted. In an ideal world it would be nice
to have an official way to grab the PyCapsules out of `__pyx_capi__`
instead of writing a new set of capsules yourself. Note that we run
into a similar problem with our own `LowLevelCallable`:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.from_cython.html#scipy.LowLevelCallable.from_cython

Getting the required name of the exported function for a fused
function requires searching through the mangled names for the one you
want.

Out of all that, my main questions are:

- Maybe I'm missing something obvious? Do people have better ideas for
improving compatibility?
- If not, would it be crazy to talk to the Cython folks about an
official method for getting the PyCapsules out of `__pyx_capi__`?

- Josh