[Cython] cython.parallel tasks, single, master, critical, barriers

[Dag Sverre Seljebotn]

On 10/12/2011 11:08 AM, Robert Bradshaw wrote:
> On Wed, Oct 12, 2011 at 1:36 AM, Dag Sverre Seljebotn
>> I wouldn't resist a builtin "channel" type in Cython (since we don't have
>> full templating/generics, it would be the only way of sending typed data
>> conveniently?).
>
> zeromq seems to be a nice level of abstraction--we could probably get
> far with a zeromq "overlay" module that didn't require the GIL. Or is
> the C API easy enough to use if we could provide convenient mechanisms
> to initialize the tasks/threads. I think perhaps the communication
> model could be solved by a library more easily than the treading
> model.

Ah, zeromq even has an in-process transport, so should work nicely for 
multithreading as well.

The main problem is that I'd like something like

ctypedef struct Msg:
     int what
     double when

cdef Msg msg
cdef channel[Msg] mychan = channel[msg](blocking=True, in_process=True)
with cython.parallel:
     ...
     if is_master():
         mychan.send(what=1, when=2.3)
     else:
         msg = mychan.recv()


Which one can't really do without either builtin support or templating 
support. One *could* implement it in C++...

C-level API just sends char* around, e.g.,

int zmq_msg_init_data (zmq_msg_t *msg, void *data, size_t size, 
zmq_free_fn *ffn, void *hint);

Dag Sverre