[Neuroimaging] parallel computation of bundle_distances_mam/mdf ?
Emanuele Olivetti
olivetti at fbk.eu
Thu Dec 15 10:43:39 EST 2016
Hi Eleftherios,
Sorry I did not mean to rush. Moreover, I am not against OpenMP at all, so
for sure collecting more information about it and OSX is valuable in any
case. I am happy Stephan is looking into that. As for my plan, I guess the
first step is to try to extend bundle_distance_*() with OpenMP, since that
is "the" meaningful choice given the code is already in Cython. I was also
considering to add a further layer (wrapper) that uses multiprocessing when
OpenMP is not available and the user requires to use multiple cores. I
already do that in my code, but I want to better understand how this fits
with the DiPy's API design.
As for using multiprocessing, I see it as a viable solution when memory is
not an issue and the code is pure Python (or when OpenMP is an issue). As
you correctly report, multiprocessing is already used in DiPy, so I was
trying to understand whether it was something deprecated or not. So from
now on I will assume it is not ;)
Best,
Emanuele
On Thu, Dec 15, 2016 at 2:54 PM, Eleftherios Garyfallidis <elef at indiana.edu>
wrote:
> Let's not be rushed to change API decisions that we have been
> investigating for years. The multiprocessing framework has its own can of
> worms. And I would personally not spend time to develop the algorithm
> twice. I mean once for multiprocessing and once for openmp. But that is
> your call.
>
> Also, let's leave Stephan a bit of time to investigate this. And let's
> also ask in Anaconda, Cython and Clang for suggestions on how to solve this
> installation problem. We need to figure out this independently of your
> contribution.
>
> OpenMP is fine with Windows and Linux. And as I said just develop it as we
> did and only temporarily in OSX you will have things single threaded.
> Because Apple decided to make radical changes we shouldn't be changing our
> protocols. Especially when they are well established standards.
>
> Yes, we use multiprocessing for the reconstruction step. But anyway you
> are not helping me here because you haven't show me your code yet. Show us
> the code! :D
>
> And please give us some days to investigate the issue.
>
> On Thu, Dec 15, 2016 at 3:42 AM Emanuele Olivetti <olivetti at fbk.eu> wrote:
>
> Thank you to all of you for the many details that help to clear up the
> picture.
>
> It is my understanding that even the last version of OSX, i.e. v10.12.*,
> does not support libomp within the Apple's Xcode development framework. At
> the same time, future version should support it, at some point. Honestly,
> this does not seem an ideal scenario, because not many OSX users chase the
> latest versions of the opeating system, while everyone has multiple cores
> since many years. For the same reason, I doubt that contacting the Xcode
> side may bring us closer to the solution.
> To Ariel: if you want to test that OpemMP works, once you build test.so,
> just import it in the Python console and execute test.test_func(). You
> should get this:
> ---
> >>> import test
> >>> test.test_func()
> Thread ID: 2
> Thread ID: 8
> Thread ID: 7
> Thread ID: 3
> Thread ID: 4
> Thread ID: 5
> Thread ID: 6
> Thread ID: 0
> Thread ID: 1
> Thread ID: 9
> ---
> If you do not enable the OpenMP support (just comment out the two related
> lines in setup.py and re-build), the output will differ and only one thread
> will show up:
> ---
> >>> import test
> >>> test.test_func()
> Thread ID: 0
> ---
>
> Anyway, what surprises me most is the failure of Anaconda's gcc+libgomp on
> OSX. In principle that toolchain is exactly the same as on GNU/Linux, so I
> see no reason for it fail. I'll try to dig more into this one.
>
> To Eleftherios: I agree with you that OpenMP is the best technical
> solution, because the use of threads - which shares memory - is definitely
> more preferable than multiprocessing - which replicates data in each
> process. Anyway, the limitations I see are still there:
> 1) OpenMP support seems problematic on non-GNU/Linux systems. By the way,
> what is the situation on Windows? Here we don't have such system so we
> couldn't try.
> 2) OpemMP works only with Cython code. This means that no Python part of
> DiPy can use parallel execution. This seems a bit too excessive to me,
> because I expect that not all parts of DiPy will be re-written in Cython,
> in the foreseeable future.
>
> So, again, I advocate for some use of the standard multiprocessing module
> :) as an temporary step when OpenMP cannot be used.
>
> Best,
>
> Emanuele
>
>
> On Wed, Dec 14, 2016 at 10:04 PM, Stephan Meesters <
> stephan.meesters at gmail.com> wrote:
>
> Hi,
>
> The instructions at http://nipy.org/dipy/installation.html#openmp-with-osx are
> outdated since the clang-omp formula does not exist anymore.
>
> With the release of Clang 3.8.0 (08 Mar 2016), OpenMP 3.1 support is
> enabled in Clang by default. You will need the -fopenmp=libomp flag while
> building.
>
> I started a while back on a DIPY homebrew formula to allow installation
> via "brew install dipy", see https://github.com/Homebrew/
> homebrew-python/pull/310. However it was based around the deprecated
> clang-omp and I didn't get around to fix it. Might look into it soon if I
> find some spare time to tweak the formula.
>
> Regards,
> Stephan
>
> 2016-12-14 17:14 GMT+01:00 Samuel St-Jean <stjeansam at gmail.com>:
>
> That also depends on which version of clang ships by default with osx, in
> which case you have to play around with it to get a new one. I think it
> starts with clang 3.7 to have openmp support (I only ever tried Mac osx
> 10.9, so anyone more experienced can chime in), but everything older has to
> go through the hombebrew gcc install and company. Might be worhtwhile to
> check if openmp support is out of the box now, and starting on which mac
> osx versions, since older ones could be problematic for first time
> installs.
>
> I also have to admit I have no idea how old is old in the mac world, so
> maybe 10.9 is already phased out by now, but it was a hard and time
> consuming building around stuff with homebrew experience (and again, first
> time I used a mac, so, I guess the average user would also have some
> issues).
>
> Samuel
>
> 2016-12-14 16:51 GMT+01:00 Eleftherios Garyfallidis <elef at indiana.edu>:
>
>
> Hi Emanuele,
>
> My understanding is that openmp was only temporarily not available when
> clang replaced gcc in osx.
>
> So, I would suggest to go ahead with openmp. Any current installation
> issues are only temporarily for osx.
> Openmp gives us a lot of capability to play with shared memory and it is a
> standard that will be around
> for very long time. Also, the great integration in cython makes the
> algorithms really easy to read.
> So, especially for this project my recommendation is to use openmp rather
> than multiprocessing. All the way! :)
>
> I am CC'ing Stephan who wrote the instructions for osx. I am sure he can
> help you with this. I would also suggest
> to check if xcode provides any new guis for enabling openmp. I remember
> there was something for that.
>
> Laterz!
> Eleftherios
>
>
>
>
> On Wed, Dec 14, 2016 at 6:29 AM Emanuele Olivetti <olivetti at fbk.eu> wrote:
>
> Hi Eleftherios,
>
> Thank you for pointing me to the MDF example. From what I see the Cython
> syntax is not complex, which is good.
>
> My only concern is the availability of OpenMP in the systems where DiPy is
> used. On a reasonably recent GNU/Linux machine it seems straightforward to
> have libgomp and the proper version of gcc. On other systems - say OSX -
> the situation is less clear to me. According to what I read here
> http://nipy.org/dipy/installation.html#openmp-with-osx
> the OSX installation steps are not meant for standard end users. Are those
> instructions updated?
> As a test of that, we've just tried to skip the steps described above and
> instead to install gcc with conda on OSX ("conda install gcc"). In the
> process, conda installed the recent gcc-4.8 with libgomp, which seems good
> news. Unfortunately, when we tried to compile a simple example of Cython
> code using parallelization (see below), the process failed (fatal error:
> limits.h : no such file or directory)...
>
> For the reasons above, I am wondering whether the very simple solution of
> using the "multiprocessing" module, available from the standard Python
> library, may be an acceptable first step towards the more efficient
> multithreading of Cython/libgomp. With "multiprocessing", there is no extra
> dependency on libgomp, or recent gcc or else. Moreover, multiprocessing
> does not require to have Cython code, because it works on plain Python too.
>
> Best,
>
> Emanuele
>
> ---- test.pyx ----
> from cython import parallel
> from libc.stdio cimport printf
>
> def test_func():
> cdef int thread_id = -1
> with nogil, parallel.parallel(num_threads=10):
> thread_id = parallel.threadid()
> printf("Thread ID: %d\n", thread_id)
> -----
>
> ----- setup.py -----
> from distutils.core import setup, Extension
> from Cython.Build import cythonize
>
> extensions = [Extension(
> "test",
> sources=["test.pyx"],
> extra_compile_args=["-fopenmp"],
> extra_link_args=["-fopenmp"]
> )]
>
> setup(
> ext_modules = cythonize(extensions)
> )
> ----
> python setup.py build_ext --inplace
>
> On Tue, Dec 13, 2016 at 11:17 PM, Eleftherios Garyfallidis <
> elef at indiana.edu> wrote:
>
> Hi Emanuele,
>
> Here is an example of how we calculated the distance matrix in parallel
> (for the MDF) using OpenMP
> https://github.com/nipy/dipy/blob/master/dipy/align/bundlemin.pyx
>
> You can just add another function that does the same using mam. It should
> be really easy to implement as we have
> already done it for the MDF for speeding up SLR.
>
> Then we need to update the bundle_distances* functions to use the parallel
> versions.
>
> I'll be happy to help you with this. Let's try to schedule some time to
> look at this together.
>
> Best regards,
> Eleftherios
>
>
> On Mon, Dec 12, 2016 at 11:16 AM Emanuele Olivetti <olivetti at fbk.eu>
> wrote:
>
> Hi,
>
> I usually compute the distance matrix between two lists of streamlines
> using bundle_distances_mam() or bundle_distances_mdf(). When the lists are
> large, it is convenient and easy to exploit the multiple cores of the CPU
> because such computation is intrinsically (embarassingly) parallel. At the
> moment I'm doing it through the multiprocessing or the joblib modules,
> because I cannot find a way directly from DiPy, at least according to what
> I see in dipy/tracking/distances.pyx . But consider that I am not
> proficient in cython.parallel.
>
> Is there a preferable way to perform such parallel computation? I plan to
> prepare a pull request in future and I'd like to be on the right track.
>
> Best,
>
> Emanuele
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20161215/8983155b/attachment-0001.html>
More information about the Neuroimaging
mailing list