[SciPy-Dev] Directory for performance analysis

[Todd]

On Wed, Oct 2, 2019 at 1:59 PM <rlucas7 at vt.edu> wrote:

> > From: Todd <toddrjen at gmail.com>
> > To: SciPy Developers List <scipy-dev at python.org>
> > Subject: [SciPy-Dev] Directory for performance analysis
> > Message-ID:
> >    <CAFpSVpJ6MoNmvvLYJJJO3dRySLHiQFhKPK=Y2dQVG1tQCjtG2Q at mail.gmail.com>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Although scipy has a directory ("benchmarks") for automatically testing
> how
> > the performance of individual functions has changed, there is no
> directory
> > as far as I have found for analysis of how particular approaches to the
> > same problem compare.  I think it would be worthwhile to have one.
> >
> > For example the "scipy.signal.convolve" function has code to
> automatically
> > choose whether to use the conventional convolution algorithm or the
> > FFT-based algorithm, based on an analysis someone did.  However, I cannot
> > find the code that actually carried out that analysis.  This is important
> > because I want to extend that analysis to include another algorithm.
> >
> > There are a bunch of other situations where particular approaches are
> > recommended as being faster in certain situations, or faster by certain
> > amounts in certain situations.
> >
> > So I think that having a directory inside the scipy source tree that
> > includes analyses like that would be worthwhile.  I was thinking
> something
> > like "benchmarks/analysis", although it could also be part of the
> > documentation proper.
>
> Hi Todd,
> This sounds interesting. The place where this could be useful to the
> releases would be in determining defaults.
>
> Is this the use you have in mind or is there another use that I’m not
> thinking of at the moment?
>
> The downside to that could be if defaults are changed they could impact
> end users of SciPy.
>

There were three situations I thought this could be useful.

The first, as you suggested, is for setting defaults.  Of course changing
defaults needs to be carefully considered.  But having the code for any
decision readily available would make such decisions easier, I think.  And
it would allow future developers to tell how current defaults were decided
on in a more reproducible way than mailing list threads.

The second is for giving advice.  In a number of places the documentation
recommends certain approaches in certain situations, even if the code
doesn't do it automatically.  This could help with that.

Third is for checking whether functions have the sort of performance
characteristics they should have.  In certain cases the relationship
between the performance of one function relative to another should be
relatively clear, even if the performance of an individual function is hard
to determine do to differences in the processor, memory, underlying
libraries, OS, etc.  This could more reliably identify errors hurting
performance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20191003/75991238/attachment.html>