[SciPy-User] peer review of scientific software

[josef.pktd at gmail.com]

On Thu, Jun 6, 2013 at 7:21 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Thu, Jun 6, 2013 at 6:23 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr> wrote:
>> On Wed, 5 Jun 2013 23:08:10 +0100
>> Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> But... have you ever sat down and written tests for a piece of widely
>>> used academic software? (Not LAPACK, but some random large package
>>> that's widely used within a field but doesn't have a comprehensive
>>> test suite of its own.) Everyone I've heard of who's done this
>>> discovers bugs all over the place. Would you personally trip over them
>>> if you didn't test the code? Who knows, maybe not. And probably most
>>> of the rest -- off by one errors here and there, maybe an incorrect
>>> normalizing constant, etc., -- end up not mattering too much. Or maybe
>>> they do. How could you even tell?
>>
>> I found bugs in scipy.ndimage.shift and in scipy.stats.linregress.
>> The first took me ages to be spotted as I was assuming the error was on
>> my side as scipy was seen as a "large library widely used".
>
> Well said.  See also Blake Griffith's current struggles with
> scipy.sparse (last message title "parametric tests, known failures and
> skipped tests").

As far as I understand these are not BUGs.
These are TDD test failures during development while adding support to
additional dtypes.
Daniel Smith was adding better indexing support, with many TDD test failures.

But that doesn't mean scipy.sparse didn't work correctly for the
initial implementation for float matrices.

I think scipy is overall in a very good shape now. Most bugs are
enhancements or refactorings and cleanup.
(Of course it doesn't mean that there are no real bugs.)

>
> If it's not tested - assume it's broken.
>
> If it's not tested and it's not broken, assume it will break soon.
>
> Don't use anything for serious work that isn't tested.
>
> At least - that has been my experience.

I agree.
But in the heavily used parts of a library, we get the bug reports by
users very fast for cases that are not covered by the unit tests.
(It took 1 to 2 years to fix all bugs in the distribution fit with
some fixed parameters, for all different combinations of fixed and not
fixed parameters.)

Josef

>
> Cheers,
>
> Matthew
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user