[SciPy-Dev] Enabling use of scipy.special from Numba

[Ralf Gommers]

On Thu, Jun 20, 2019 at 6:01 AM Joshua Wilson <josh.craig.wilson at gmail.com>
wrote:

> Hey everyone,
>
> For a while I've been thinking about how we can improve the
> compatibility of `scipy.special` with Numba. There is a way to do that
> currently by using `cython_special`:
>
> https://github.com/numba/numba/issues/3086
>
> but that has a few drawbacks:
>
> (1) it's not very user friendly
> (2) it relies on the (I think?) private implementation details of
> Cython's `__pyx_capi__` (including the name mangling scheme used for
> fused functions)
>

Is there a difference on our side between cython_special and
cython_blas/lapack? Because Numba already interfaces with those to
accelerate numpy.linalg. We'd like Numba to understand scipy.linalg too
(don't think it does, but didn't check recently), and ideally all those
interfaces would work the same way.

Cheers,
Ralf


(3) it's clumsy for complex functions
>
> I think (3) requires changes on the Numba side, e.g.
>
> https://github.com/numba/numba/issues/3860
>
> For (1) you could imagine us implementing another Numba specific API,
> but that runs into some immediate issues:
>
> - We'd need Numba as a dependency (which has been discussed:
> https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html)
> - We don't want to just keep implementing new APIs
>
> For those reasons I'd prefer to find another solution. In an ideal
> world I'd like to solve (2) in such a way that it's easy for a
> third-party library to wrap `cython_special` in Numba jitted
> functions. There is a solution to that already that doesn't involve
> `__pyx_capi__`: a third-party library writes some Cython glue that
> exports PyCapsules of function pointers to the specializations of all
> the `cython_special` functions and then you grab those using ctypes to
> construct the jitted functions. For the Cython side of that see the
> docs on specialization:
>
>
> http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-specializations
>
> But that feels rather convoluted. In an ideal world it would be nice
> to have an official way to grab the PyCapsules out of `__pyx_capi__`
> instead of writing a new set of capsules yourself. Note that we run
> into a similar problem with our own `LowLevelCallable`:
>
>
> https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable.from_cython.html#scipy.LowLevelCallable.from_cython
>
> Getting the required name of the exported function for a fused
> function requires searching through the mangled names for the one you
> want.
>
> Out of all that, my main questions are:
>
> - Maybe I'm missing something obvious? Do people have better ideas for
> improving compatibility?
> - If not, would it be crazy to talk to the Cython folks about an
> official method for getting the PyCapsules out of `__pyx_capi__`?
>
> - Josh
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190620/77490b35/attachment.html>