[Neuroimaging] [DIPY] Setting up a platform for offline end-to-end quality assurance for DIPY

[Eleftherios Garyfallidis]

Dear Matthew, Maxime, Ariel and all,

Mr. Dumont and I have started creating some workflows which can be run by
the command line. These are made to work with large real datasets.

I think it would be great if we could use a different type of testing from
what we were using right now. Most of the testing we use is actually fast
testing of functions and we should definitely continue having that.

But I think we need also an end-to-end offline testing where we actually
test with big whole brain datasets and then we can collect some automatic
quality assurance reports. In that way we cover most of unexpected issues.

Now, the problem with having such a platform is that it needs computing
power and some disk space. It may need a descent computer to run for 24
hours for example and let's say around 100 GBytes of free disk space. Then
it will also need to send some automated reports to say that is all good or
not.

Ariel has suggested  to use the cloud and docker but I am afraid that it
will be too expensive for our pockets right now except if someone can
donate to the project.

An alternative idea would be to go gradually and setup one of the computers
in Sherbrooke or in Berkeley or in Seattle to do such a job. I think this
QA should run once/twice a week rather than every day.

Now there are other platforms that need to run relatively frequently. One
is the examples for the documentation and then there is Omar's validation
framework which actually needs a large cluster. We can deal with those at a
later stage.

The easiest way forward with the workflows that I see right now is that Mr.
Dumont adds a script in dipy/tools that will run all the workflows as we do
with make_examples.py that run all the examples. We first try this platform
in Sherbrooke and then we need to figure out a way to send automated
reports to the core developers or to berkeley builders and so on.
Maybe sending a PDF or HTML of the output screenshots would be also a good
idea.

Let me know what you think.

Cheers,
Eleftherios
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20160303/17927815/attachment.html>