[Cython] OpenMP support

[mark florisson]

On 11 March 2011 01:46, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted <faltet at pytables.org> wrote:
>> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigué:
>>> mark florisson, 08.03.2011 18:00:
>>> > What I meant was that the
>>> > wrapper returned by the decorator would have to call the closure
>>> > for every iteration, which introduces function call overhead.
>>> >
>>> >[...]
>>> >
>>> > I guess we just have to establish what we want to do: do we
>>> > want to support code with Python objects (and exceptions etc), or
>>> > just C code written in Cython?
>>>
>>> I like the approach that Sturla mentioned: using closures to
>>> implement worker threads. I think that's very pythonic. You could do
>>> something like this, for example:
>>>
>>>      def worker():
>>>          for item in queue:
>>>              with nogil:
>>>                  do_stuff(item)
>>>
>>>      queue.extend(work_items)
>>>      start_threads(worker, count)
>>>
>>> Note that the queue is only needed to tell the thread what to work
>>> on. A lot of things can be shared over the closure. So the queue may
>>> not even be required in many cases.
>>
>> I like this approach too.  I suppose that you will need to annotate the
>> items so that they are not Python objects, no?  Something like:
>>
>>     def worker():
>>         cdef int item  # tell that item is not a Python object!
>>         for item in queue:
>>             with nogil:
>>                 do_stuff(item)
>>
>>     queue.extend(work_items)
>>     start_threads(worker, count)
>
> On a slightly higher level, are we just trying to use OpenMP from
> Cython, or are we trying to build it into the language? If the former,
> it may make sense to stick closer than one might otherwise be tempted
> in terms of API to the underlying C to leverage the existing
> documentation. A library with a more Pythonic interface could perhaps
> be written on top of that. Alternatively, if we're building it into
> Cython itself, I'd it might be worth modeling it after the
> multiprocessing module (though I understand it would be implemented
> with threads), which I think is a decent enough model for managing
> embarrassingly parallel operations. The above code is similar to that,
> though I'd prefer the for loop implicit rather than as part of the
> worker method (or at least as an argument). If we went this route,
> what are the advantages of using OpenMP over, say, pthreads in the
> background? (And could the latter be done with just a library + some
> fancy GIL specifications?) One thing that's nice about OpenMP as
> implemented in C is that the serial code looks almost exactly like the
> parallel code; the code at http://wiki.cython.org/enhancements/openmp
> has this property too.

Indeed, and this works very nicely for for loops because they already
constitute blocks. However, if you want to support other OpenMP-like
functionality (and not saying that we want to, but if), then you have
two options:

1) you need a way to explicitly declare blocks in Cython, or
2) you need to write all your blocks as functions and call those,
instead (most likely quite bothersome).

If we want an interface like multiprocessing, I think we can just
existing batteries, of which just one example is the functionality
described in PEP 3148 (concurrent.futures, new in 3.2). People would
only have to write Python functions that could release the GIL if
needed.

> Also, I like the idea of being able to hold the GIL by the invoking
> thread and having the "sharing" threads do the appropriate locking
> among themselves when needed if possible, e.g. for exception raising.
>
> Another thought I had is, there might be other usecases for being able
> to emit generic pragmas statements, how far would that get us?

I gave that, but more generally, a 'verbatim' clause some thought.
However, syntax would be bothersome. I thought about e.g.

cdef verbatim:
     C code here

Access to variables (i.e., substitute Cython variable names with their
name-mangled counterparts) might be provided through e.g. '$var'
syntax, as used by string.Template. But I can already hear people
object rather loudly, so I rejected the thought :). If, of course,
access to variables is not allowed but instead we allow only simple
pragmas, a simple cython.pragma("#pragma bla") should probably
suffice.

On the other hand, do we really need to mangle (non-temporary)
variable names? Global variables are declared static and variables may
be redeclared in inner C scopes.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>