[Cython] [cython-users] Segfault in PyThread_release_lock

[mark florisson]

On Monday, 15 August 2011, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 08/15/2011 11:42 AM, mark florisson wrote:
>
> On 15 August 2011 02:22, Nikolaus Rath<Nikolaus at rath.org>  wrote:
>
> mark florisson<markflorisson88 at gmail.com>  writes:
>
> On 14 August 2011 17:50, Nikolaus Rath<Nikolaus at rath.org>  wrote:
>
> mark florisson<markflorisson88 at gmail.com>  writes:
>
> On 12 August 2011 20:23, Nikolaus Rath<Nikolaus at rath.org>  wrote:
>
> Hello,
>
> The following segfault is completely incomprehensible to me:
>
> (gdb) bt full
> #0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
> No locals.
> #1  0x000000000050e3e7 in PyThread_release_lock (lock=0x0) at ../Python/thread_pthread.h:374
>         thelock = 0x0
>         status = 0
>         error = 0
> #2  0x00000000004c4f7b in PyEval_ReleaseLock () at ../Python/ceval.c:262
> No locals.
> #3  0x00000000004fbeef in PyThreadState_DeleteCurrent () at ../Python/pystate.c:319
>         tstate = 0x925da0
> #4  0x00000000004fc6a3 in PyGILState_Release (oldstate=PyGILState_UNLOCKED)
>     at ../Python/pystate.c:651
>         tcur = 0x925da0
>         __PRETTY_FUNCTION__ = "PyGILState_Release"
> [...]
> Why am I getting an error so deep in the python internals?
>
>
> The error is deep because Python needs to do certain things. So what
> it's doing here is destructing your thread state, which means that was
> the last call to PyGILState_Ensure(). This means your function was
> called from another thread than a Python thread. It looks to me like
> the GIL is not initialized (__pyx_gilstate_save =
> PyGILState_UNLOCKED). Are you sure you have called
> PyEval_EvalInitThreads()?
>
>
> No, I have not. But isn't Cython supposed to do this automatically when
> entering the function if I define the function as "with gil"?
>
>
> No, it acquires the GIL, but it does not attempt to initialize it.
> Seeing how many people have issues with it, I'm thinking that perhaps
> we should if the module contains with gil functions, or with gil
> blocks in nogil functions. If you're using cython.parallel and with
> gil blocks, the GIL will be initialized at module-load time. Note that
> initializing the GIL slows down serially executed Python code, so
> that's why Cython doesn't initialize it unconditionally. It will have
> to be called at module-load time, as it should be called in the main
> python thread (or from any other python thread started by the
> thread(ing) module).
>
>
> Oh, wow. That's a tricky one. And it explains why I have the segfault
> only in a reduced testcase, but not in the full blown application (the
> application probably creates a Python thread before calling into Cython
> code). I banged my head against the wall for quite some time already
> because of that.
>
> Thanks a lot! It would certainly have cost me a *lot* of time if'd had
> to figure that out myself.
>
> I wouldn't mind if Cython inserted the call automatically, but a big
> warning in
> http://docs.cython.org/src/userguide/external_C_code.html#acquiring-and-releasing-the-gil
> (under the "acquiring the GIL") might already be enough.
>
>
> Best,
>
>    -Nikolaus
>
> --
>   »Time flies like an arrow, fruit flies like a Banana.«
>
>   PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C
>
>
>
> Definitely, I'll do that one of these days. I do think it's rather
> common to have with gil functions called as a callback from another
> (non-python) thread. I'm not so sure about with gil blocks in nogil
> functions.
>
> @Cython-dev: Do we merely want to update the docs, or do we want to
> initialize the GIL for either case, or only for the with gil
> functions? I'm not entirely sure what the overhead is for
> single-threaded code, but I'd say we need to initialize it for both
> cases.
>
>
> I'm not sure if I like magic related to having used "with gil" on a function signature -- it's a bit too magic and obscure.

I don't know, you'll be 100% safe and there's not any really big
drawback (if used for either case).
> I think we should have another directive for this as well as the #ifdef macro (that is, the directive controls the default value of the C macro), and document that a bit better.
>
> As for the default value of that, here's two proposals:
>
>  Alt. 1) Always initialize the GIL by default.
>
>  Alt. 2) Issue a warning whenever a with-gil function is present unless the directive is explicitly set (one way or the other). Perhaps use the same warning if "with nogil" is used as well.

'with nogil' is not a problem, releasing and acquiring in the main
thread is safe.

> I'm in favour of 1), since it would save a lot of frustration among users, since alt 2) is a rather minor case only costing a little bit of performance (?), and since in many of the cases where the GIL should not be initialized, the multithreading support is somewhat likely to turned off at the CPython level anyway during compilation (?).
I don't think a lot of users compile without threading support. But
you don't need to initialize the gil if you don't use with gil blocks
in nogil functions or with gil functions, so why always initialize it?
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>