[Cython] 'with gil:' statement

[mark florisson]

On 16 March 2011 16:14, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 16.03.2011 11:28:
>>
>> I implemented the 'with gil:' statement
>> [...]
>> The 'with gil:' statement can now be used in the same way as 'with
>> nogil:'. Exceptions raised from GIL blocks will be propagated if
>> possible (i.e., if the return type is 'object'). Otherwise it will
>> jump to the end of the function and use the usual
>> __Pyx_WriteUnraisable, there's not really anything new there.
>
> I'm not sure if this is a good idea. "nogil" blocks don't have a way to
> handle exceptions, so simply jumping out of them because an inner 'with gil'
> block raised an exception can have unexpected side effects.
>
> Would you do the same when calling a cdef function that uses "with gil" in
> its signature? We don't currently do that, but it feels like it's the same
> situation.
>

I beg to differ. Consider this code:

with nogil:
    with gil:
        raise SomeError
    printf("something\n")

With normal 'with' statements the printf (in the snippet above) would
be skipped, and so it is with 'with gil:' blocks, it follows the exact
semantics of the Python behaviour as you expect it. With statements
are like try/finally, and the 'with (no)gil:' blocks are implemented
as such. The above would translate to:

try:
   release gil
   try:
       acquire gil
       raise SomeError
   finally:
       release gil
   printf("something\n")
finally:
    acquire gil

If users need to do (C-level) cleanup, they need to wrap the code in
the 'with gil:' in something like try/finally, or perhaps try/except.

In fact, if you declare a function 'with gil', you will want to either
handle any exception in that function, or you want your caller to
check for errors while returning an error indicator. So in that case,
you should still do error checking, if you don't, exceptions will
simply be ignored.

Another related issue, you cannot declare your function with both
'with gil' and  'except NULL' at the same time, as calling such a
function without having the GIL will segfault your program. In fact, I
think we should add a check for this, as e.g. the following code will
segfault in func() when it calls __Pyx_WriteUnraisable, as it will try
to get the exception from a NULL thread-state.

from cpython.ref cimport PyObject
from libc.stdio cimport printf

cdef PyObject *func2() except NULL with gil:
    raise Exception("blah")

cdef void func() nogil:
    cdef PyObject *p = func2()
    printf("This is not printed!\n")

func()

Of course, you cannot really declare the return type of func2 as
'object', because you wouldn't be able to call it from 'nogil'
sections that way.

I guess my point is, if the user isn't very careful about the way
exceptions are handled, undesired behaviour may occur. With both
functions and with gil blocks, the user will have to override the
default behaviour if needed. I think "python semantics" most
accurately reflect the syntax for 'with gil:' blocks.

>> For functions declared 'nogil' that contain 'with gil:' statements, it
>> will safely lock around around the initialization of any Python
>> objects and set up the refnanny context (with appropriate preprocessor
>> guards). At the end of the function it will safely lock around the
>> teardown of the refnanny context. With 'safely' I mean that it will
>> create a thread state if it was not already created and may be called
>> even while the GIL is already held (using PyGILState_Ensure()). This
>> means variables are declared and initialized in the same way as in
>> normal GIL-holding functions (except that there is additional
>> locking), and of course the GIL-checking code ensures that errors are
>> issued if those variables are attempted to be used outside any GIL
>> blocks.
>
> I find that surprising semantics. So the GIL will always be acquired at
> least twice in the following example, regardless of the input?
>
>    cdef int callme(bint flag) nogil:
>         if flag:
>             with gil: x = object()
>

Yes. It should be noted to users that 'with gil:' does not come
without its repercussions.

>> Could someone review the patch (which is attached)? Maybe check if I
>> haven't missed any side cases and such?
>
> From a first look, the test file you added seems far too short. I would
> expect that this feature requires a lot more testing in combination with
> declared and undeclared local variables, type inference, several exception
> raising and catching situations (e.g. raise in one block, catch in an outer
> block, try-finally, ...) or looping. It may also have an impact on Vitja's
> control flow analysis branch that's worth considering.

I agree. I think I'll start a branch and work on some more tests.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>