[Cython] Calling gil-requiring function not allowed without gil

[Robert Bradshaw]

On Wed, Aug 17, 2011 at 1:08 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/17/2011 09:12 AM, Robert Bradshaw wrote:
>> So your proposal is that "with cython.synchronized" has the same
>> effect as a Python operation, in that its a compile time error to do
>> it while not holding the gil? (As opposed to "with gil" which actively
>> acquires the lock (which is more similar to java synchronized)?) Would
>
> Yes, the former. It does not acquire a lock under curreent CPython.
>
> But the intention is very much to prepare the ground for
> IronPython/Jython/GIL-less CPython/autogil-in-Cython too. Basically a tool
> to protect & document logic that depends on GIL to avoid races, so that it
> is future-safe.
>
> So in Cython-for-Jython it would have to acquire a lock, and be more like
> Java synchronized.

I think a with gil statement (always acquiring the lock) has more
consistent semantics and accomplishes the same goal. Is "with gil"
expensive if one already has the gil? I'm assuming here that the gil
is re-entrant (or could be made to see so).

>> this cause the function to have a GIL requiring/acquiring signature? I
>
> Not in isolation.
>
> But, IF (after a long deprecation cycle) we start to infer "nogil" on
> function signatures, then the presence of this Python statement would stop
> nogil from getting inferred.

Knowing the subtleties of whether any Python operations are used,
especially as the specific optimizations done continue to evolve, this
seems like a dangerous thing to infer.

> The intention is simply: If you have a cdef function which uses no Python
> operations, but still relies on holding the GIL to avoid race conditions
> (exactly the situation Stefan described initially), then you can use
> cython.synchronized to avoid any future inference from removing the GIL.

Again, with gil, if acceptable from a gil-holding context, seems
sufficient. Then if one did such inference, one could isolate it to
the important block.

>> suppose the reason that one implicitly holds the gil is that it allows
>> you to do arbitrary Python operations by default. Having to explicitly
>> mark when one wants to hold the GIL would be a big regression for
>> applications that do a lot of Python interaction (e.g. wrappers).
>
> Yes, I'm not proposing that at all.
>
>> I'm still thinking it could be reasonable for extern functions to be
>> nogil by default, as a user of C libraries is expected to manually and
>> externally respect its threading constraints. (This would require a
>> long deprecation period.)
>
> +1.

Actually, it's relatively safe to switch, as it will only make
currently illegal code legal, but won't break existing code.

> Perhaps this would be safer if we introduce "expects gil" (explicit
> declaration for neither "nogil" or "with gil"), and then start to emit
> warnings if a GIL-mode isn't declared on cdef functions which has their
> address taken.
>
> Thus, if you pass a callback to a C library, you'll get a reminder to
> declare a GIL mode.

That's a nice idea. I have to admit that all these special gil
declarations are a bit messy. I'd also rather introduce clear
decorators, e.g.

@cython.requires_gil  # expects gil
cdef a(): ...

@cython.requires.gil(False) # nogil
cdef b(): ...

@cython.aquires_gil  # with gil
cdef c(): ...

(Actually, now that we have the "with gil" statement, it could be
worth considering simply noticing the pattern of the entire function
body in a with gil block/as the first statement and acquiring the GIL
before argument parsing.)

Note that we need to declare functions as requiring the GIL to allow
for declaring cpython.pxd if extern functions are implicitly nogil.

- Robert