[SciPy-Dev] Backwards Compatibility for low level LAPACK routines

[josef.pktd at gmail.com]

On Thu, Aug 2, 2018 at 2:37 PM, Ilhan Polat <ilhanpolat at gmail.com> wrote:
> Due their historical evolution, there are certain LAPACK wrappers that are
> not standardized. Some work with minimum lwork variables instead of their
> optimal values. Also these routines often return quite big arrays during the
> lwork queries, to demonstrate :
>
> import scipy.linalg as la
> la.lapack.dgeqrf(a=np.random.rand(400,400), lwork=-1)
>
> is a workspace size query (via lwork=-1). The current default size is
> "3*a.shape[0] + 1" hence 1201. However the optimal workspace size is 12800
> on my machine. Therefore the mismatch is sometimes quite dramatic especially
> in some other routines. Notice also that to obtain this number the routine
> actually returns a 400-long array tau and requires the input matrix to be
> transferred back and forth. Moreover, they can't be handled via
> scipy.linalg.lapack._compute_lwork function.
>
> There are a few routines like this and I feel like they should be fixed and
> I'm willing to. However this means that their output signature is going to
> change which imply backwards compatibility breaks. I tried to see whether we
> could deprecate them with new wrappers with modified names, but to be
> honest, that would create too many duplicates. On the other hand I don't
> have a feeling of how much break this would mean out there in the wild.
>
> Is this break an acceptable one or not? (well, none is acceptable
> preferably, but in despair...)
>
> Any other alternatives, thoughts are most welcome.

Is this only for the python functions or also for the cython wrappers to LAPACK?

Binary incompatibilities are pretty painful. We just got rid of the
compatibility with scipy's old cython wrapper code in statsmodels.
Both distributing binaries and conditional compilation depending on
the installed scipy versions is fragile and there doesn't seem to be
good packaging support for it.

Josef



>
> best,
> ilhan
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>