[SciPy-Dev] merging optional Pythran support soon

[Ralf Gommers]

On Mon, Dec 28, 2020 at 4:02 PM Stanley Seibert <sseibert at anaconda.com>
wrote:

> On Sat, Dec 26, 2020 at 3:33 PM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>
>> And to preempt the obvious question: no we don't need to compare with
>> Numba. That situation didn't change from last time we discussed it; Numba
>> is a heavy and fragile runtime dependency, and supporting libraries like
>> SciPy isn't Numba's core focus. I also checked in with Stan Seibert (Numba
>> core dev) recently, and he agreed with that assessment.
>>
>
> Just to jump in here, I would say that supporting SciPy, specifically,
> isn't Numba's *current* core focus.  As one of the most core PyData
> libraries (second only to NumPy), we agree that one needs to be very
> conservative about introducing new code and new ways of doing things to
> SciPy, and Numba's design approach makes it not a direct drop-in for Cython
> use cases.  Pythran fits more naturally into a Cython usage pattern, which
> is beneficial here.  Clearly Numba would need to have more robust
> ahead-of-time compilation support to be usable in a library like SciPy, and
> that is still on the back burner while we think about various issues.
>

> However, other libraries that do numerical computing like SciPy (but do
> not have the constraints of SciPy) *are* Numba's focus.  I just wanted to
> make sure there was no confusion about this.  :)
>

Thanks Stan, that's all helpful context!


> As a meta comment, this PR is basically implementing a sort of dependency
> injection for the SciPy internals, to allow a different compiler system to
> be swapped in to compile a specific internal function.
>

If you mean the conditional compile based on a SCIPY_USE_PYTHRAN
environment variable, we don't want to keep that part long-term. It's just
an escape hatch during the introduction period in case of issues, and we
had the Cython code already so it was easy to do. Maintaining two
implementations in parallel is usually a bad idea.

Where this could be generalized is very interesting, and relevant if there
> is a future where a number of SciPy functions could be compiled by one of
> two compilers.  For example, if the type information embedded in a comment
> for Pythran here:
>
>
> https://github.com/scipy/scipy/pull/8306/files#diff-6e0de4105e10b6c609d5d18639757bc58716f165d14f677f5fcca8bf57edf805R6
>
> were available at runtime and/or in a more readily parsible form, that
> would be part of opening up SciPy to more compiler tools.  I'm not sure if
> a best practice has emerged for writing Python type annotations with NumPy
> types, though.
>

Yes, that is a great point. NumPy 1.20 will be the first release to include
type annotations. Annotating ndarray properties like shape and dtype is
complicated though and still WIP - initial support landed very recently in
https://github.com/numpy/numpy/pull/17719.

Also note that Transonic aims to use type annotations and then allow using
Cython, Pythran and Numba as backends:
https://fluiddyn.netlify.app/transonic-vision.html.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20201228/4dcc87b6/attachment.html>