[Python-ideas] Suggestion for a new thread API for Python (OpenMP inspired)

[Jesse Noller]

On Thu, Jan 22, 2009 at 5:53 AM, Sturla Molden <sturla at molden.no> wrote:
>
> I have lately been using OpenMP to write parallel C and Fortran code. I must
> admit I am impressed. OpenMP is a much better abstraction for writing
> concurrent code than using Win32/posix threads directly (or similarly
> threading.Thread i Java or Python). What matters most is that code can be
> written as sequential, tested, and then parallelised using compiler pragmas.
> This is much easier than writing code intended to be parallel from the
> start.
>
> Not only is the abstraction more easy to apply, it also leads to fewer
> problems with deadlocks, race conditions, livelocks, etc.
>
> I was thinking something similar could be created for Python, e.g. on top of
> the existing thread or threading modules, and possibly multiprocessing. I
> believe a context manager could be used for this purpose. What I have in
> mind is an API that would look approximately like this (OpenMP pragmas for C
> on top, proposed Python equivalent below):
>
>
> #pragma omp parallel
> with pymp.Pool() as pool:
>
> #pragma omp for
> for item in pool.parallel(<iterable>):
>
> #pragma omp for shedule(guided)
> for item in pool.parallel(<iterable>, shed='guided'):
>
> #pragma omp parallel for
> with pymp.Pool() as pool:
>   for item in pool.parallel(<iterable>):
>
> #pragma omp barrier
> pool.barrier()
>
> #pragma omp section
> pool.section(fun, *args, **kwargs)
>
> #pragma omp parallel sections
> with pymp.Pool() as pool:
>   pool.section(fun1, *args, **kwargs)
>   pool.section(fun2, *args, **kwargs)
>
> #pragma omp master
> if pool.master:
>
> #pragma omp critical
> #pragma omp atomic
> with pool.lock:
>
> #pragma omp single
> with pool.single():
>
> #pragma omp ordered
> with pool.ordered():
>
>
> This is all trivial to program, except for the context manager on top. It
> has somehow to get access to the code block below, spawn multiple threads,
> and execute that block in each of the threads. I am not sure how to grab the
> next executable block as a Python object (so I could pass it to eval or
> exec), so a little help would be appreciated :)
>

Hi Sturla,

 Interesting that you bring this up - while I'm not in the know about
openMP - I have been sketching out some improvements to threading and
multiprocessing that follow some of this thinking.

The first batch would be adding context managers where appropriate to
the multiprocessing module (e.g. pool and managers/etc). The second is
adding a library of decorators where you could say:

@multiprocessing.Pool(5, 'apply')
def myfunc()
  ....

Or something along those lines. In the case of getting the function
object back, decorators make more sense for some of the use cases.

Now, for threading - I personally feel that there's some improvements
to be made by adding a series of abstractions and utilities on top of
threading ala multiprocessing.pool and the like. See
java.util.concurrent for some good examples of threading abstractions.

The same decorators and contextmanagers could be used for any
threading abstractions added.

Just some thoughts - right now I've sworn off writing new features
until I zero out the multiprocessing bug queue, so these might be done
in 2025.

-jesse