[SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU

[Neil Godber]

Hi Ralf,

Thanks for your email and happy to assist in whatever way I can. I've
reached out to Dr Xiaoyei Li of Lawrence Berkley who appears to be the
principal maintainer of the code base with a request to host the _mt code
on github. Given the other two versions are already hosted I can't see that
being an issue. I've also queried whether developments are done in sync and
if so, if there is a pending patching for mt in the similar fashion to
sequential.

I will revert with further information as I receive it.

Cheers,
Neil

On Mon, 24 May 2021 at 04:33, Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Fri, May 21, 2021 at 2:11 AM Neil Godber <njgodber at gmail.com> wrote:
>
>> Hi dev-list,
>>
>> Further last, realised that I failed to link to the issue itself despite
>> linking to virtually everything else:
>>
>> Replace superLU sequential with superLU dist · Issue #14096 · scipy/scipy
>> (github.com) <https://github.com/scipy/scipy/issues/14096>
>>
>> Cheers,
>> Neil
>>
>> On Fri, 21 May 2021 at 10:04, Neil Godber <njgodber at gmail.com> wrote:
>>
>>> Hi dev-list,
>>>
>>
> Thanks for the proposal Neil.
>
>
>>> I've logged an enhancement request on github to bring the
>>> parallel versions of SuperLU into scipy. As requested I'm circulating the
>>> enhancement here to solicit comment/feedback/engagement.
>>>
>>> Content of request is as follows (with light editing and links):
>>>
>>> "Is your feature request related to a problem? Please describe.
>>>
>>> The direct solution of sparse matrices is a common problem that arises
>>> across many domains. Current scipy.sparse.linalg provides spsolve and splu
>>> to solve such classes of problems. Neither solution appears to utilise more
>>> than one or two threads leaving a lot of the available compute on modern
>>> machines unused. Other solutions (MUMPS, Pardiso) exploit multicore
>>> processors but are either minimally supported and/or locked behind
>>> proprietary libraries.
>>>
>>> Describe the solution you'd like
>>> Fortunately SuperLU has two multithreaded implementations superLU_mt and
>>> superLU_dt, both actively maintained (SuperLU: Home Page (nersc.gov)
>>> <https://portal.nersc.gov/project/sparse/superlu/>). Ideally SciPy
>>> would replace the current sequential implementation with one of the
>>> parallel implementations to yield large performance improvements.
>>>
>> The relevant one seems to be superLU_mt. The `_dt` flavor is for
> distributed computing, which is out of scope for SciPy. superLU_mt says it
> has both pthreads and OpenMP interfaces - we'd prefer pthreads (and
> actually forbid OpenMP within SciPy).
>
> The last release of superLU_mt is v3.1, from March 2016. Our current copy
> of SuperLU is 5.2.1, from May 2016 - but then there's a bunch of patches
> applied to it, and then https://github.com/scipy/scipy/pull/12243 did a
> sync with SuperLU master in May 2020. So it would be necessary to figure
> out if we can get superLU_mt from upstream master as well, and that we
> don't lose bug fixes we applied since 2016. Could you look into that Neil?
> And do you know if the parallel and sequential versions are developed in
> sync or not?
>
> Cheers,
> Ralf
>
>
> Describe alternatives you've considered
>>> pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL routines
>>> (github.com) <https://github.com/dwfmarchant/pyMKL>) - MKL with issues
>>> associated with that for non Intel processors, unmaintained relatively to
>>> scipy, PyTrilinos (implements all three superLU types but only available on
>>> osx and linux PyTrilinos | Trilinos
>>> <https://trilinos.github.io/pytrilinos.html>), petsc (petsc4py PETSc /
>>> petsc · GitLab <https://gitlab.com/petsc/petsc>) (implements all three
>>> superLU types but only available on osx and linux). All are relatively
>>> inaccessible compared to ubiquity and accessibility of scipy.
>>>
>>> Additional context (e.g. screenshots)
>>> Anecdotally large number of projects use various non scipy sparse
>>> solvers to achieve high level of performance. This comes with considerable
>>> time and maintenance investment. It would be fantastic for scipy to provide
>>> a 'performance competitive' sparse direct solver out of the hood as it does
>>> for dense matrices."
>>>
>>>
>>> Cheers,
>>>
>>> Neil
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210524/7fa8fd38/attachment.html>