[Numpy-discussion] Accelerate your Python code with parallel processing

[Ronnie Hoogerwerf]

Matthieu, linking in a multi-threading enabled library to numpy, like  
MKL, will provide speedup to only a limited set of operations and  
scale only on when you limit yourself to the cores in a single smp- 
like system. Star-P on the other hand, since it is MPI based, scales  
on cluster and covers a much larger set of operations that can be  
performed in parallel plus it provides a simple mechanism to do  
embarrassingly parallel operations.

Ronnie

On Jun 29, 2007, at 1:46 PM, Matthieu Brucher wrote:

> Hi,
>
> Is there a comparison with parallel libraries that can be branched  
> on numpy like MKL ? (and IPP for random number ?)
>
> Matthieu
>
> 2007/6/29, Ronnie Hoogerwerf <  
> rhoogerwerf at interactivesupercomputing.com>:
> I am an Application Engineer at Interactive Supercomputing and we  
> are rolling out a beta version of our Star-P product for Python. We  
> are actively looking for computationally intensive Python  
> application to port to Star-P. Star-P is a parallel application  
> development platform that allows users to tap into the power and  
> memory of supercomputers from the comfort of the favorite desktop  
> applications, in this case Python.
>
> Star-P is capable of both fine-grained parallel computation and  
> embarrassingly parallel computation. The fine-grained mode of our  
> Star-P Python implementation has been modeled on the Python NumPy  
> package - for example:
>
> x = starp.random.rand (20000,20000)
> y = starp.linalg.inv(x)
>
> instead of
>
> x = numpy.random.rand(20000,20000)
> y = numpy.linalg.inv(x)
>
> Where the first couple of lines are executed on the Star-P parallel  
> server in full C/MPI mode and the last couple of lines are executed  
> on the desktop using Python.
>
> The embarrassingly parallel mode is capable of executing any Python  
> module, although input and output parameters are currently limited  
> to NumPy arrays, scalars, and strings - for example:
>
> y = starp.ppeval(mymodule.dosomething,x)
>
> instead of
>
> for i in range(0,n):
>     y[:,:,i] = mymodule.dosomething(x[:,i])
>
> Where again in the former example the iterations are spread out  
> over the available CPUs (note the abstraction - user need not worry  
> regarding the number of CPUs) on the Star-P server using Python and  
> in the latter the looping is doing in serial on the client using  
> Python.
>
> We are looking for real Python application that you would be  
> willing to share with us that  we can port to Star-P. We want to  
> use this experience as a basis for further improvements and  
> development of our Python client.
>
> Thanks,
> Ronnie
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070629/950d21c4/attachment.html>