[SciPy-User] peer review of scientific software

[Matthew Brett]

Hi,

On Thu, Jun 6, 2013 at 6:23 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr> wrote:
> On Wed, 5 Jun 2013 23:08:10 +0100
> Nathaniel Smith <njs at pobox.com> wrote:
>
>> But... have you ever sat down and written tests for a piece of widely
>> used academic software? (Not LAPACK, but some random large package
>> that's widely used within a field but doesn't have a comprehensive
>> test suite of its own.) Everyone I've heard of who's done this
>> discovers bugs all over the place. Would you personally trip over them
>> if you didn't test the code? Who knows, maybe not. And probably most
>> of the rest -- off by one errors here and there, maybe an incorrect
>> normalizing constant, etc., -- end up not mattering too much. Or maybe
>> they do. How could you even tell?
>
> I found bugs in scipy.ndimage.shift and in scipy.stats.linregress.
> The first took me ages to be spotted as I was assuming the error was on
> my side as scipy was seen as a "large library widely used".

Well said.  See also Blake Griffith's current struggles with
scipy.sparse (last message title "parametric tests, known failures and
skipped tests").

If it's not tested - assume it's broken.

If it's not tested and it's not broken, assume it will break soon.

Don't use anything for serious work that isn't tested.

At least - that has been my experience.

Cheers,

Matthew