[SciPy-user] Multithreading cookbook entry

[Anne Archibald]

On 21/02/2008, Anand Patil <anand.prabhakar.patil at gmail.com> wrote:
> On Thu, Feb 21, 2008 at 8:30 AM, Anne Archibald
>  <peridot.faceted at gmail.com> wrote:
>
>  > It would be useful to go through the code looking at which ones
>  >  do and don't release the GIL, and put it in their docstrings; it might
>  >  be possible to make more release the GIL. It's a pretty safe bet that
>  >  the ufuncs do; I would guess that the linear algebra functions do too.
>  >  Probably not much else.
>
>
> I second that suggestion. In fact I'd be willing to help out if it's a
>  tedious but simple job.

Well, what needs to happen is that someone needs to go through and
track down occurrences of Py_BEGIN_ALLOW_THREADS ..
Py_END_ALLOW_THREADS in numpy.

A brute-force way of finding code that probably doesn't do it would be
to simply run each function in a foreach() with two threads and then
with one and see if there's any speedup. Messy and crude; probably
better just to look at the code, but numpy can be labyrinthine.

> Makes sense. Does anyone know about f2py extensions with 'cf2py
>  threadsafe' set? From the f2py user's guide, the threadsafe option
>  will
>
>  Use Py_BEGIN_ALLOW_THREADS .. Py_END_ALLOW_THREADS block around the
>  call to Fortran/C function.
>
>  Is that sufficien to release the GIL? What if the functions have callbacks??

That's exactly what is needed to release the GIL.

I think, from looking at the code, that F2PY does nothing to reacquire
the GIL if it's entering a callback; this would mean that using
callbacks in a "threadsafe" function would cause a crash. So Don't Do
That. (But I'm not totally sure; maybe try generating one just to
check.)

Anne