[Cython] 'with gil:' statement

[Dag Sverre Seljebotn]

On 03/17/2011 09:27 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 17.03.2011 08:38:
>> On 03/17/2011 12:24 AM, Greg Ewing wrote:
>>> Stefan Behnel wrote:
>>>
>>>> I'm not sure if this is a good idea. "nogil" blocks don't have a 
>>>> way to
>>>> handle exceptions, so simply jumping out of them because an inner 
>>>> 'with
>>>> gil' block raised an exception can have unexpected side effects.
>>>
>>> Seems to me that the __Pyx_WriteUnraisable should be done at
>>> the end of the 'with gil' block, and execution should then
>>> continue from there.
>>>
>>> In other words, the effect on exception handling should be
>>> the same as if the 'with gil' block had been factored out into
>>> a separate function having no exception return value.
>>>
>>
>> -1.
>>
>> I consider the fact that exceptions don't propagate from some 
>> functions a
>> "currently unfixable bug". We should plan for it being fixed some day.
>
> It can't be fixed in general, because there are cases where exceptions 
> simply cannot be propagated. Think of C callbacks, for example. C 
> doesn't have a "normal" way of dealing with exceptions, so if an 
> exception that originated from a callback simply leads to returning 
> from the function, it may mean that the outer C code will simply 
> continue to execute normally. Nothing's won in that case.

Yes, that's a good point. (This is what I used setjmp/longjmp to work 
around BTW, to longjmp across the calling Fortran code. I knew it wasn't 
doing any mallocs/frees, let alone any file handling etc., so this was 
safe.)

I'll admit that I'm mostly focused on code like

def f():
     with nogil:
         for ...:
             A
             if something_exceptional:
                 with gil:
                     raise Exception(...)
             B
         C

where I'd say it's up to me to make sure that B and C can safely be 
skipped. It would be a major pain to have my raised exception here be 
"trapped" -- in fact, it would make the "with gil" statement unusable 
for my purposes.



>
> In code:
>
>     cdef void c_callback(...) nogil:
>         ... do some C stuff ...
>         with gil:
>             ... do some Python stuff ...
>         ... do some more C stuff ...
>
> So far, there are two proposed ways of doing this.
>
> 1) acquire the GIL on entry and exit, handling unraisable exceptions 
> right before exiting.
>
> 2) keep all GIL requiring code inside of the "with gil" block, 
> including unraisable exceptions.
>
> I find (2) a *lot* more intuitive, as well as much safer. We can't 
> know what effects the surrounding "do C stuff" code has. It may 
> contain thread-safe C level cleanup code for the "with gil" block, for 
> example, or preparation code that enables returning into the calling C 
> code. Simply jumping out of the GIL block without executing the 
> trailing code may simply not work at all.

I think you find (2) more intuitive because you have a very detailed 
knowledge of Cython and CPython, but that somebody new to Cython would 
expect a "with" statement to have the same control flow logic as the 
Python with statement. Of course, I don't have any data for that.

How about this compromise: We balk on the code you wrote with:

Error line 345: Exceptions propagating from "with gil" block cannot be 
propagated out of function, please insert try/except and handle exception

So that we require this:

with gil:
     try:
         ...
     except:
         warnings.warning(...) # or even cython.unraisable(e)

This keeps me happy about not abusing the with statement for strange 
control flow, and makes the "with gil" useful for raising exceptions 
inside regular def functions with nogil blocks.


>
>
>> We could perhaps fix exception propagation from nogil functions by using
>> some conventions + setjmp/longjmp. Mono does this when calling into 
>> native
>> code, and I recently did it manually in Cython to propagate exceptions
>> through the Fortran wrappers in SciPy.
>
> Regardless of the topic of this thread, it would be nice to have 
> longjmp support in Cython. Lupa, my Cython wrapper for LuaJIT, 
> currently has to work around several quirks in that area.

Not sure what you mean here, I used longjmp (in a function without any 
Python objects) and it seems to work just fine. Did I miss anything?

>
>
>> Also, the GIL may not be around
>> forever even in CPython? (All arguments I've seen for keeping it has 
>> been
>> along the lines of "it slows down serial code", not that it is 
>> considered a
>> good thing.)
>
> If it ever gets removed, there will surely have to be an emulation 
> layer for C modules. Many of them simply use it as thread-lock, and 
> that's totally reasonable IMHO.

Good point. But there may be an option to disable said emulation layer 
that we want to make use of in Cython...

(This is relevant today for Cython-on-.NET, for instance.)

>
>
>> Designing a language around the GIL feels like a dead-end to me.
>
> We keep having diverging opinions about the GIL. I like it, and I keep 
> repeating myself by saying that "threading should be explicit". Having 
> a way to lock the whole interpreter and to keep parallel execution and 
> reentry points to well defined places in your code is a great feature.

I think all of this just comes from using Cython for totally different 
things, which gives us different perspectives on the GIL. I guess we 
should just sit down in Munich and look at each other's codes and see if 
we can understand one another that way. A big discussion on whether "the 
GIL is good or not" is not very constructive.

Dag Sverre