[SciPy-Dev] Enhancement Request: SuperLU_dist/SuperLU_mt replace current scipy.sparse.superLU

[Neil Godber]

Hi Ralf,

Saw your comment on the issue. Is there anything else I can do to help?
Unfortunately my C/C++ skills are lacking so I'm not sure how much help I
on the actual integration itself.

Cheers,
Neil

On Sat, 29 May 2021 at 17:00, Neil Godber <njgodber at gmail.com> wrote:

> Hi Ralf + ScipyDev,
>
> Dr Li has helpfully uploaded the SuperLU_mt code base to github
> https://github.com/xiaoyeli/superlu_mt . Dr Li indicated that most
> development effort is focused on _dt, but that sequential and mt remain
> supported and bug reports will be attended to.
>
> Cheers,
> Neil
>
> On Mon, 24 May 2021 at 11:09, Neil Godber <njgodber at gmail.com> wrote:
>
>> Hi Ralf,
>>
>> Thanks for your email and happy to assist in whatever way I can. I've
>> reached out to Dr Xiaoyei Li of Lawrence Berkley who appears to be the
>> principal maintainer of the code base with a request to host the _mt code
>> on github. Given the other two versions are already hosted I can't see that
>> being an issue. I've also queried whether developments are done in sync and
>> if so, if there is a pending patching for mt in the similar fashion to
>> sequential.
>>
>> I will revert with further information as I receive it.
>>
>> Cheers,
>> Neil
>>
>> On Mon, 24 May 2021 at 04:33, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Fri, May 21, 2021 at 2:11 AM Neil Godber <njgodber at gmail.com> wrote:
>>>
>>>> Hi dev-list,
>>>>
>>>> Further last, realised that I failed to link to the issue itself
>>>> despite linking to virtually everything else:
>>>>
>>>> Replace superLU sequential with superLU dist · Issue #14096 ·
>>>> scipy/scipy (github.com) <https://github.com/scipy/scipy/issues/14096>
>>>>
>>>> Cheers,
>>>> Neil
>>>>
>>>> On Fri, 21 May 2021 at 10:04, Neil Godber <njgodber at gmail.com> wrote:
>>>>
>>>>> Hi dev-list,
>>>>>
>>>>
>>> Thanks for the proposal Neil.
>>>
>>>
>>>>> I've logged an enhancement request on github to bring the
>>>>> parallel versions of SuperLU into scipy. As requested I'm circulating the
>>>>> enhancement here to solicit comment/feedback/engagement.
>>>>>
>>>>> Content of request is as follows (with light editing and links):
>>>>>
>>>>> "Is your feature request related to a problem? Please describe.
>>>>>
>>>>> The direct solution of sparse matrices is a common problem that arises
>>>>> across many domains. Current scipy.sparse.linalg provides spsolve and splu
>>>>> to solve such classes of problems. Neither solution appears to utilise more
>>>>> than one or two threads leaving a lot of the available compute on modern
>>>>> machines unused. Other solutions (MUMPS, Pardiso) exploit multicore
>>>>> processors but are either minimally supported and/or locked behind
>>>>> proprietary libraries.
>>>>>
>>>>> Describe the solution you'd like
>>>>> Fortunately SuperLU has two multithreaded implementations superLU_mt
>>>>> and superLU_dt, both actively maintained (SuperLU: Home Page
>>>>> (nersc.gov) <https://portal.nersc.gov/project/sparse/superlu/>).
>>>>> Ideally SciPy would replace the current sequential implementation with one
>>>>> of the parallel implementations to yield large performance improvements.
>>>>>
>>>> The relevant one seems to be superLU_mt. The `_dt` flavor is for
>>> distributed computing, which is out of scope for SciPy. superLU_mt says it
>>> has both pthreads and OpenMP interfaces - we'd prefer pthreads (and
>>> actually forbid OpenMP within SciPy).
>>>
>>> The last release of superLU_mt is v3.1, from March 2016. Our current
>>> copy of SuperLU is 5.2.1, from May 2016 - but then there's a bunch of
>>> patches applied to it, and then
>>> https://github.com/scipy/scipy/pull/12243 did a sync with SuperLU
>>> master in May 2020. So it would be necessary to figure out if we can get
>>> superLU_mt from upstream master as well, and that we don't lose bug fixes
>>> we applied since 2016. Could you look into that Neil? And do you know if
>>> the parallel and sequential versions are developed in sync or not?
>>>
>>> Cheers,
>>> Ralf
>>>
>>>
>>> Describe alternatives you've considered
>>>>> pyMKL (Pardiso dwfmarchant/pyMKL: Python wrappers to Intel MKL
>>>>> routines (github.com) <https://github.com/dwfmarchant/pyMKL>) - MKL
>>>>> with issues associated with that for non Intel processors, unmaintained
>>>>> relatively to scipy, PyTrilinos (implements all three superLU types but
>>>>> only available on osx and linux PyTrilinos | Trilinos
>>>>> <https://trilinos.github.io/pytrilinos.html>), petsc (petsc4py PETSc
>>>>> / petsc · GitLab <https://gitlab.com/petsc/petsc>) (implements all
>>>>> three superLU types but only available on osx and linux). All are
>>>>> relatively inaccessible compared to ubiquity and accessibility of scipy.
>>>>>
>>>>> Additional context (e.g. screenshots)
>>>>> Anecdotally large number of projects use various non scipy sparse
>>>>> solvers to achieve high level of performance. This comes with considerable
>>>>> time and maintenance investment. It would be fantastic for scipy to provide
>>>>> a 'performance competitive' sparse direct solver out of the hood as it does
>>>>> for dense matrices."
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Neil
>>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210530/bcf0fac3/attachment.html>