[SciPy-user] SciPy, MPI and OpenMP

[Matthieu Brucher]

2008/8/18 Lorenzo Isella <lorenzo.isella at gmail.com>:
> Dear All,
> I have recently attended a crash course on MPI and OpenMP. The
> examples always involved C or Fortran code.
> Now, I have a thought: if working on a single processor, I hardly need
> to use pure C or pure Fortran. I usually write a Fortran code for the
> bottlenecks and compile it with f2py to create a python module I then
> import.
> Hence two questions:
> (1) Can I do something similar with many processors? E.g.: write a
> Python code, embed some compiled Fortran code which is supposed to run
> on many processors, get the results and come back to Python.

Of course, this is possible. You can use OpenMP easily by using an
OpenMP compatible compiler, and for MPI, there are several packages
that can be used (search through the archives, one of them standas
apparently out ;))

> Python--->Fortran on many processors--->back to Python.
> (2)Is it also possible to directly parallelize a Python code? I heard
> about thread locking in Python.

The GIL is a lock on the interpreter. If your code is mainly C code,
it can release the GIL and therefore you can run parallel code in
Python. Numpy releases the lock whenever it can. If you use SWIG, you
only have to add -threads to the arguments and it releases the GIL
(there are several features in SWIG allowing you to tune the GIL
processing, I've explained them on my blog).

> I did some online research, there seems to be a lot of projects trying
> to combine Python and MPI/OpenMP, but many look rather "experimental".
> In particular, of course, I would like to hear about SciPy and
> parallel computing.

You can always use the processing module try to mimic the threading
module, but using processes. For MPI, search the archives, like I
said, there are some interesting posts.

Cheers,
Matthieu
-- 
French PhD student
Website : http://matthieu-brucher.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn : http://www.linkedin.com/in/matthieubrucher