PyGilState_Ensure interrupts python critical sections

[Chris Mellon]

On Nov 12, 2007 6:56 AM,  <billy.omahony at gmail.com> wrote:
> Hi,
>
> I have a native windows thread in a c python module which calls into
> python code and adds an item to a data structure (a home-grown
> circular buffer). At the same time my main python application is
> removing items from this data structure.
>
> Unlike native python containers adding and removing items from the
> circular buffer is not atomic. So I need to put access to it in a
> critical section. Using threading.Lock will work for native python
> threads competing for access. But when access is from a windows thread
> this will not work. That is because my call to PyGilState_ensure will
> preempt my native python thread ***even when it is inside the critical
> section***.
>
> What is going on looks something like this (I think).
>
>
>    Py      Python       Windows       Py       threading.Lock
> resource
>   Sched    Thread       Thread       Code            |              |
>     |                     |            |             |              |
>     |Go (GIL)#            |            |             |              |
>     |        #            |            |             |              |
>     |        #            |            |             |              |
>     |        #...Doit.....|...........>|             |              |
>     |        #            |            |. acquire...>|              |
>     |<-PyGILState_Ensure--|            |             |              |
>     |       ...          ...          ...           ...
>     |Stop    #
>     |-------`|
>     |
>     |----Ensure rtns-----># PyObject_  |                            |
>              |   :        |CallMethod  |             |              |
>              |   :        |.(Doit)...> |. acquire...>| DEADLOCK     |
>                  :
>                  :
>                  :.how does python thread tell
>                     PyScheduler not to give away
>                       Gil until we are done with
>                           critical section??
>
> So my question is how in python do I tell the scheduler not to prempt
> the current python thread. Especially how to tell it not to give the
> GIL to any calls to PyGILState_ensure until further notice (i.e. until
> I am outside my critical section?? It sounds like a reasonable request
> - surely this exists already but I can't find it.
>

It took me some time to understand the problem you were describing
because you've got some terms backward - there's no Python scheduler,
and calling PyGILState_ensure doesn't preempt anything. The Python
interpreter *releases* the GIL every so often to allow other threads
looking for it to run, but calling the python GIL functions has no
effect on preemption.

The problem is that the GIL is being released while your object lock
is held, a second thread (started from C) acquires the GIL and then
blocks on the object lock. What you seem to be seeing is that it
blocking on the object lock is preventing it from releasing the GIL,
which prevents the python thread from running and releasing the lock.

This shouldn't happen - blocking on a lock releases the GIL, so the
python thread should run, release the GIL, and eventually your C
thread should be able to acquire both locks at the same time. Are you
sure that you're correctly acquiring the GIL in your C code?


The data flow you *should* be seeing should look something like this:

GIL     object lock (P=Python, C=C, *=released)
-------------------
P       P           Python holds both locks
*       P           Python releases the GIL, inside critical section
C       P           C thread acquires the GIL and starts executing Python code
*       P           C thread tries to acquire object lock and blocks,
releasing the GIL
P       P           Python thread re-acquires the GIL
P       *           Python thread exits critical section and releases
the object lock
*       *           Python thread releases the GIL (can be in any
order with next state)
*       C           The C thread acquires the object lock and blocks on the GIL
C       C           C thread acquires the GIL and continues execution.

> One thing that may work (though the documentation does not
> specifically say so) is using setcheckinterval() to set the check
> interval to a very large value, acessing my shared structure and then
> setting the check interval back to the previous value. Provided my
> access to the shared structure takes less byte codes than what I set
> checkinterval to I should be okay. However that would be very
> dependant on the exact fine detail of how the check interval works and
> may not be compatible with other Python releases
>
> Maybe someone familiar with the python source code would know for
> sure?
>
> I am using Python 2.4.3 on windows XP.
>
> Thanks for any help/suggestions offered!
> BR,
> Billy.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>