[Cython] 'with gil:' statement

[mark florisson]

On 17 March 2011 09:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 17.03.2011 08:38:
>>
>> On 03/17/2011 12:24 AM, Greg Ewing wrote:
>>>
>>> Stefan Behnel wrote:
>>>
>>>> I'm not sure if this is a good idea. "nogil" blocks don't have a way to
>>>> handle exceptions, so simply jumping out of them because an inner 'with
>>>> gil' block raised an exception can have unexpected side effects.
>>>
>>> Seems to me that the __Pyx_WriteUnraisable should be done at
>>> the end of the 'with gil' block, and execution should then
>>> continue from there.
>>>
>>> In other words, the effect on exception handling should be
>>> the same as if the 'with gil' block had been factored out into
>>> a separate function having no exception return value.
>>>
>>
>> -1.
>>
>> I consider the fact that exceptions don't propagate from some functions a
>> "currently unfixable bug". We should plan for it being fixed some day.
>
> It can't be fixed in general, because there are cases where exceptions
> simply cannot be propagated. Think of C callbacks, for example. C doesn't
> have a "normal" way of dealing with exceptions, so if an exception that
> originated from a callback simply leads to returning from the function, it
> may mean that the outer C code will simply continue to execute normally.
> Nothing's won in that case.
>
> In code:
>
>    cdef void c_callback(...) nogil:
>        ... do some C stuff ...
>        with gil:
>            ... do some Python stuff ...
>        ... do some more C stuff ...
>
> So far, there are two proposed ways of doing this.
>
> 1) acquire the GIL on entry and exit, handling unraisable exceptions right
> before exiting.
>
> 2) keep all GIL requiring code inside of the "with gil" block, including
> unraisable exceptions.
>
> I find (2) a *lot* more intuitive, as well as much safer. We can't know what
> effects the surrounding "do C stuff" code has. It may contain thread-safe C
> level cleanup code for the "with gil" block, for example, or preparation
> code that enables returning into the calling C code. Simply jumping out of
> the GIL block without executing the trailing code may simply not work at
> all.

Which is exactly why users will have to handle exceptions if they want
their C code to execute.

>
>> We could perhaps fix exception propagation from nogil functions by using
>> some conventions + setjmp/longjmp. Mono does this when calling into native
>> code, and I recently did it manually in Cython to propagate exceptions
>> through the Fortran wrappers in SciPy.
>
> Regardless of the topic of this thread, it would be nice to have longjmp
> support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work
> around several quirks in that area.
>
>
>> Also, the GIL may not be around
>> forever even in CPython? (All arguments I've seen for keeping it has been
>> along the lines of "it slows down serial code", not that it is considered
>> a
>> good thing.)
>
> If it ever gets removed, there will surely have to be an emulation layer for
> C modules. Many of them simply use it as thread-lock, and that's totally
> reasonable IMHO.
>
>
>> Designing a language around the GIL feels like a dead-end to me.
>
> We keep having diverging opinions about the GIL. I like it, and I keep
> repeating myself by saying that "threading should be explicit". Having a way
> to lock the whole interpreter and to keep parallel execution and reentry
> points to well defined places in your code is a great feature.
>
>
>> I'm OK
>> with being practical in the face of the limitations of today; but let's
>> keep "with gil" and "with nogil" something that can become noops in the
>> future without too much pain. Yes, I know that if the GIL goes it will
>> break Stefan's lxml code, and I'm sure other code -- I'm just saying that
>> we shouldn't make the language design even more GIL-centric than it
>> already
>> is.
>
> It's not. Even a removal of the GIL won't remove the fact that C can't
> propagate exceptions.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>