[Cython] Acquisition counted cdef classes

[Dag Sverre Seljebotn]

On 10/25/2011 06:58 PM, mark florisson wrote:
> On 25 October 2011 12:22, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> The problem is not so much the INCREF (which is just an indirect add), it's
>> the DECREF, which contains a conditional jump based on an unknown external
>> value, that may trigger external code. That can kill several C compiler
>> optimisations for the surrounding code. (And that would only get worse by
>> using a dedicated locking mechanism.)

What you could do is a form of psuedo-garbage-collection where, when the 
Cython refcount/acquisition count reaches 0, you enqueue a Python DECREF 
until you're holding the GIL anyway. If sticking it into the queue is 
unlikely(), and it is transparent to the compiler that it doesn't 
dispatch into unknown code.

(And regarding Stefan's comment about Erlang: It's all about available 
libraries. A language for concurrent computing running on CPython and 
able to use all the libraries available for CPython would be awesome. It 
doesn't need to be named Cython -- show me an Erlang port to the CPython 
platform and I'd perhaps jump ship.)


> Anyway, sorry for the long mail. I agree this is likely not feasible
> to implement, although I would like the functionality to be there.
> Perhaps I'm trying to solve problems which don't really need to be
> solved. Maybe we should just use multiprocessing, or MPI and numpy
> with global arrays and pickling. Maybe memoryviews could help out with
> that as well.

Nice conclusion. I think prange was a very nice 80%-there-solution 
(which is also the way we framed it when starting), but the GIL just 
creates to many barriers. Real garbage collection is needed, and CPython 
just isn't there.

What I'd like to see personally is:

  - A convenient utility to allocate an array in shared memory, so that 
when you pickle a view of it and send it to another Python process with 
multiprocessing and it unpickles, it gets a slice into to the same 
shared memory. People already do this but it's just a lot of jumping 
through hoops. A good place would probably be in NumPy.

  - Decent message passing using ZeroMQ in Cython code without any 
Python overhead, for fine-grained communication in Cython code in Python 
processes spawned using multiprocessing. I think this requires some 
syntax candy in Cython to feel natural enough, but perhaps it can be put 
on a form so that it is not ZeroMQ-specific.

Dag Sverre