[SciPy-Dev] Enabling use of scipy.special from Numba

[Stanley Seibert]

(Numba dev here...)

I don't think SciPy would need to take Numba as a dependency to solve this,
any more than cffi or ctypes does.  All we need on the Numba side is a
standard way, given a scipy.special function object to obtain the
underlying C function pointer and the C type signature (basic types, no
Python objects) of that function.  If there is a standard set of attributes
for us to pull that information from, the adapter can live inside the Numba
code base and SciPy needs minimal changes.


On Wed, Jun 19, 2019 at 11:02 PM Joshua Wilson <josh.craig.wilson at gmail.com>
wrote:

> Hey everyone,
>
> For a while I've been thinking about how we can improve the
> compatibility of `scipy.special` with Numba. There is a way to do that
> currently by using `cython_special`:
>
> https://github.com/numba/numba/issues/3086
>
> but that has a few drawbacks:
>
> (1) it's not very user friendly
> (2) it relies on the (I think?) private implementation details of
> Cython's `__pyx_capi__` (including the name mangling scheme used for
> fused functions)
> (3) it's clumsy for complex functions
>
> I think (3) requires changes on the Numba side, e.g.
>
> https://github.com/numba/numba/issues/3860
>
> For (1) you could imagine us implementing another Numba specific API,
> but that runs into some immediate issues:
>
> - We'd need Numba as a dependency (which has been discussed:
> https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html)
> - We don't want to just keep implementing new APIs
>
> For those reasons I'd prefer to find another solution. In an ideal
> world I'd like to solve (2) in such a way that it's easy for a
> third-party library to wrap `cython_special` in Numba jitted
> functions. There is a solution to that already that doesn't involve
> `__pyx_capi__`: a third-party library writes some Cython glue that
> exports PyCapsules of function pointers to the specializations of all
> the `cython_special` functions and then you grab those using ctypes to
> construct the jitted functions. For the Cython side of that see the
> docs on specialization:
>
>
> http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-specializations
>
> But that feels rather convoluted. In an ideal world it would be nice
> to have an official way to grab the PyCapsules out of `__pyx_capi__`
> instead of writing a new set of capsules yourself. Note that we run
> into a similar problem with our own `LowLevelCallable`:
>
>
> https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.from_cython.html#scipy.LowLevelCallable.from_cython
>
> Getting the required name of the exported function for a fused
> function requires searching through the mangled names for the one you
> want.
>
> Out of all that, my main questions are:
>
> - Maybe I'm missing something obvious? Do people have better ideas for
> improving compatibility?
> - If not, would it be crazy to talk to the Cython folks about an
> official method for getting the PyCapsules out of `__pyx_capi__`?
>
> - Josh
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190620/ceb78b83/attachment.html>