[SciPy-user] SciPy, MPI and OpenMP

[Robert Kern]

On Mon, Aug 18, 2008 at 10:00, Lorenzo Isella <lorenzo.isella at gmail.com> wrote:
> Dear All,
> I have recently attended a crash course on MPI and OpenMP. The
> examples always involved C or Fortran code.
> Now, I have a thought: if working on a single processor, I hardly need
> to use pure C or pure Fortran. I usually write a Fortran code for the
> bottlenecks and compile it with f2py to create a python module I then
> import.
> Hence two questions:
> (1) Can I do something similar with many processors? E.g.: write a
> Python code, embed some compiled Fortran code which is supposed to run
> on many processors, get the results and come back to Python.
>
> Python--->Fortran on many processors--->back to Python.
> (2)Is it also possible to directly parallelize a Python code? I heard
> about thread locking in Python.

There is a global interpreter lock (GIL) when touching Python data
structures. If you handle the threads entirely in the Fortran code and
never call back into Python until you are finished with the threads,
this should not be an issue for you.

In bad ASCII art:

Python  |Fortran /------\   |  Python
--------|-------<-------->--|--------
        |        \------/   |

> I did some online research, there seems to be a lot of projects trying
> to combine Python and MPI/OpenMP, but many look rather "experimental".
> In particular, of course, I would like to hear about SciPy and
> parallel computing.

Most of the MPI wrappers these days are fairly mature. I think some of
the OpenMP work is still pretty new, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco