[SciPy-User] Multiprocessing and shared memory

[Sturla Molden]

Felix Schlesinger skrev:
>
>> with nogil:
>>    <suite>
>>     
>
> Well this only works if the parallel operation does not touch any
> python objects, right? Otherwise it would reenter the interpreter
> without the GIL and not be thread-safe anymore.
Yes and no. Cython will tell you if the GIL is required (you'll get an 
compilation error).

You can use numpy ndarrays in nogil blocks.

import numpy as np
cimport numpy as np

def foobar():
   
    cdef int n
    cdef double rvalue = 0.1

    cdef np.ndarray[double, ndim=1, mode='c'] array
   
    array = np.zeros(10)
  
    with nogil:

        for n from 0 <= n < 10:
            array[n] = rvalue

http://wiki.cython.org/tutorials/numpy


>  In other words the
> whole algorithm would have to be moved into cython. 
Only the worst bottlenecks.


> Is this approach
> stable in your experience? 
Cython is a stabile approach. It is used by SciPy and Sage, and many 
other projects. It will probably be used to port NumPy to Python 3. It's 
not a toy.

You can prototype your code in Python. When you are done, you can add in 
some type declarations here and there, compile, and get the same 
performance as C or C++.

> It sounds like it would be easy to create
> race-conditions to me.
In my experience it is easier to get race conditions when using threads 
in C, C++ or Java. Anything outside nogil blocks is serialized with 
Cython. Since Cython is compiled, there are no thread switch in 
Cython-land util you release the GIL manually. Anything not declared 
nogil becomes thread-safe by default.

Also: You can run closures as threads in Python. Just put the loops you 
want parallelized in a closure, and you have the same more or less the 
same constructs as used by OpenMP or Apple's GCD.