[Neuroimaging] [DIPY] Setting up a platform for offline end-to-end quality assurance for DIPY

[Eleftherios Garyfallidis]

The problem with using CircleCI is the memory restriction. I think it is
4GB. Please correct me if wrong.
Just some of the tractography files we load is of this size. Even for
running our doc examples we need more than 8 GB.

We need a machine that has 12-16GB of RAM. Or we need to refactor some of
the methods that require larger amounts of memory to use
memmaps or similar. But still 4GB will be a strong restriction. We are
often dealing with big datasets.

Ariel, as I said in my previous e-mail at this stage we need a machine to
process for 24 hours continuously and do that 1 to 2 times a week. But of
course if there are important
issues we will need to do that sooner. Also it would be good to have some
multiprocessing involved. The machine used should have multiple cores if
possible.
So, what do you think will be the logistics for that?

Also we need a system which will allow to report and send feedback after
the execution of the workflows. In a way Matthew has already setup some of
these things
in the Berkeley bots. I want to hear his opinion too on this.

I also like the idea of having a docker that can be moved anywhere.






On Thu, Mar 3, 2016 at 2:26 PM, vanessa sochat <vsochat at stanford.edu> wrote:

> +1
>
> On Thu, Mar 3, 2016 at 11:23 AM, Chris Filo Gorgolewski <
> krzysztof.gorgolewski at gmail.com> wrote:
>
>> Ah yes that makes sense. You can however split your tests (for example
>> one workflow per line) and easily run a battery of tests that takes more
>> than 2h.
>> On Mar 3, 2016 11:16 AM, "vanessa sochat" <vsochat at stanford.edu> wrote:
>>
>>> The upper limit for each test is 2 hours (120 minutes).
>>>
>>> On Thu, Mar 3, 2016 at 11:02 AM, Chris Filo Gorgolewski <
>>> krzysztof.gorgolewski at gmail.com> wrote:
>>>
>>>> Yes, longer jobs (AFAIK there is no limit as long as the job is
>>>> printing something every 10 minutes) and more RAM (4GB) are the main
>>>> advantages.
>>>> On Mar 3, 2016 10:46 AM, "Yaroslav Halchenko" <lists at onerussian.com>
>>>> wrote:
>>>>
>>>>>
>>>>> On Thu, 03 Mar 2016, Chris Filo Gorgolewski wrote:
>>>>>
>>>>> > Have a look at waht we are doing for Nipype on CircleCI (on the free
>>>>> open
>>>>> > source tier):
>>>>>
>>>>> > https://github.com/nipy/nipype/blob/master/circle.yml
>>>>> > https://circleci.com/gh/nipy/nipype
>>>>>
>>>>> > All of the workflows we run for tests take over 3h to finish.
>>>>> Similar set
>>>>> > up is implemented in nilearn project.
>>>>>
>>>>> sorry for ignorance -- haven't used circleci yet.  So is that the
>>>>> crucial advantage of circle ci (over travis) that it allows for
>>>>> longer jobs and more caching space or what?
>>>>>
>>>>> --
>>>>> Yaroslav O. Halchenko
>>>>> Center for Open Neuroscience     http://centerforopenneuroscience.org
>>>>> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
>>>>> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
>>>>> WWW:   http://www.linkedin.com/in/yarik
>>>>> _______________________________________________
>>>>> Neuroimaging mailing list
>>>>> Neuroimaging at python.org
>>>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>>>
>>>>
>>>> _______________________________________________
>>>> Neuroimaging mailing list
>>>> Neuroimaging at python.org
>>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>>
>>>>
>>>
>>>
>>> --
>>> Vanessa Villamia Sochat
>>> Stanford University
>>> (603) 321-0676
>>>
>>> _______________________________________________
>>> Neuroimaging mailing list
>>> Neuroimaging at python.org
>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>
>>>
>> _______________________________________________
>> Neuroimaging mailing list
>> Neuroimaging at python.org
>> https://mail.python.org/mailman/listinfo/neuroimaging
>>
>>
>
>
> --
> Vanessa Villamia Sochat
> Stanford University
> (603) 321-0676
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20160303/4fcc6ca4/attachment-0001.html>