[New-bugs-announce] [issue34416] PyCapsule_Import seems to release the GIL without acquiring it

[Paul Ganssle]

New submission from Paul Ganssle <p.ganssle at gmail.com>:

I was recently debugging some multithreaded Rust code that was deadlocking, and I tracked it down to what I'm fairly certain is a bug somewhere in PyCapsule_Import, where it seems that PyCapsule_Import releases the GIL without first acquiring it.

I've attached a MWE of a multi-threaded application that is able to simultaneously acquire the GIL twice. The relevant portion is here:

void *acquire_gil(void *arg) {
    bool import = ((arg_t *)arg)->import;
    int n = ((arg_t *)arg)->id;

    printf("Waiting for GIL (%d)\n", n);
    int gstate = PyGILState_Ensure();
    printf("Gil acquired! (%d)\n", n);
    usleep(125000);
    if (import) {
        PyCapsule_Import(CAPSULE_NAME, 0);
    }
    usleep(125000);
    PyGILState_Release(gstate);
    printf("Gil released! (%d)\n", n);
    return NULL;
}

If you run it with `./gil` such that the PyCapsule_Import call is never reached, you get:

    Waiting for GIL (0)
    Gil acquired! (0)
    Waiting for GIL (1)
    Gil released! (0)
    Gil acquired! (1)
    Gil released! (1)

However, if you run with `./gil import` such that PyCapsule_Import is reached, you get (emphasis mine):

    Waiting for GIL (0)
    Gil acquired! (0)
    Waiting for GIL (1)
    **Gil acquired! (1)**
    **Gil released! (1)**
    Gil released! (0)

For convenience sake, I have created a small repo with a make file for the PoC: https://github.com/pganssle/capsule-gil-poc

I have tested this on version 3.6.6 and 3.7.0. The makefile works in a virtualenv, but you have to manually tweak the PY_MAJOR and PY_MINOR variables in Makefile because I didn't get too fancy with it.

----------
files: main.c
messages: 323620
nosy: p-ganssle, pablogsal
priority: normal
severity: normal
status: open
title: PyCapsule_Import seems to release the GIL without acquiring it
versions: Python 3.6, Python 3.7
Added file: https://bugs.python.org/file47753/main.c

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34416>
_______________________________________