[SciPy-user] Multithreading cookbook entry

[Young, Karl]

Sorry for the huge picture, really ignorant question (my specialty !) but why couldn't all this be encapsulated in something like a version of MPI for python (since I don't know anything about the GIL I assume that might be the answer) - I recall years ago that Sun had a beautiful design (amazing !) re. their version of MPI that ran on their NUMA architecture - the library allowed the code to know whether one was communicating/running in shared and/or distributed memory and dispatched things appropriately (i.e. threaded or not). I know this is a bit futuristic but all of this should just be transparent to the non expert user; one should only have to learn details when parallelization is non-trivial. Maybe MPI isn't the best model for consolidation but it seems to abstract parallel operations in a reasonable way.    

Karl Young
Center for Imaging of Neurodegenerative Disease, UCSF
VA Medical Center, MRS Unit (114M)
Phone:  (415) 221-4810 x3114
FAX:    (415) 668-2864
Email:  karl young at ucsf edu



-----Original Message-----
From: scipy-user-bounces at scipy.org on behalf of Anand Patil
Sent: Thu 2/21/2008 11:39 AM
To: SciPy Users List
Subject: Re: [SciPy-user] Multithreading cookbook entry
 
>  Yep. Oops. Fixed in the v2 versions of the files. The wiki doesn't
>  make a very good version control system. Is it worth incorporating
>  those files into scipy?

I vote yes. In my opinion the following would combine to form a killer feature:

- The handythread idea is developed a little, maybe to provide
functionality comparable to OpenMP
- Instructions for releasing the GIL in different extension types
(swig, f2py, pyrex) are combined in one place
- The numpy functions that release the GIL are clearly enumerated.

Seriously, this is too big of a deal to be just a cookbook entry. I
spent a full week last month beating my head against OpenMP trying to
do something embarrassingly parallel in an f2py extension. I had to
apply a patch to gcc 4.2's libgomp, compile it manually, learn how
linking works, and try several other options because OpenMP was so
frustrating. Now it works but I have tons of bug-prone code
duplication in Fortran because I couldn't figure out how to just apply
the same parallelism structure to all subroutines.  The ability to
multithread from Python would have saved me all of that work.

Anand
_______________________________________________
SciPy-user mailing list
SciPy-user at scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-user