[Cython] OpenMP support

[Stefan Behnel]

Francesc Alted, 08.03.2011 20:16:
> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigué:
>> mark florisson, 08.03.2011 18:00:
>>> What I meant was that the
>>> wrapper returned by the decorator would have to call the closure
>>> for every iteration, which introduces function call overhead.
>>>
>>> [...]
>>>
>>> I guess we just have to establish what we want to do: do we
>>> want to support code with Python objects (and exceptions etc), or
>>> just C code written in Cython?
>>
>> I like the approach that Sturla mentioned: using closures to
>> implement worker threads. I think that's very pythonic. You could do
>> something like this, for example:
>>
>>       def worker():
>>           for item in queue:
>>               with nogil:
>>                   do_stuff(item)
>>
>>       queue.extend(work_items)
>>       start_threads(worker, count)
>>
>> Note that the queue is only needed to tell the thread what to work
>> on. A lot of things can be shared over the closure. So the queue may
>> not even be required in many cases.
>
> I like this approach too.  I suppose that you will need to annotate the
> items so that they are not Python objects, no?  Something like:
>
>       def worker():
>           cdef int item  # tell that item is not a Python object!
>           for item in queue:
>               with nogil:
>                   do_stuff(item)
>
>       queue.extend(work_items)
>       start_threads(worker, count)

You *can* use Python references inside of nogil blocks, even if there's a 
lot of stuff that you can't do with them, such as looking up Python 
attributes, calling them, or even assigning their reference to another 
Python variable. But you can access cdef attributes on them, for example, 
and some simple conversions to C types may also work (AFAIR). But you're 
generally right that passing it into a function (as in the example above) 
would likely not work. In any case, Cython will tell you if you try to do 
anything that's not allowed.

Stefan