PyGilState_Ensure interrupts python critical sections

[billy.omahony at gmail.com]

Hi,

I have a native windows thread in a c python module which calls into
python code and adds an item to a data structure (a home-grown
circular buffer). At the same time my main python application is
removing items from this data structure.

Unlike native python containers adding and removing items from the
circular buffer is not atomic. So I need to put access to it in a
critical section. Using threading.Lock will work for native python
threads competing for access. But when access is from a windows thread
this will not work. That is because my call to PyGilState_ensure will
preempt my native python thread ***even when it is inside the critical
section***.

What is going on looks something like this (I think).


   Py      Python       Windows       Py       threading.Lock
resource
  Sched    Thread       Thread       Code            |              |
    |                     |            |             |              |
    |Go (GIL)#            |            |             |              |
    |        #            |            |             |              |
    |        #            |            |             |              |
    |        #...Doit.....|...........>|             |              |
    |        #            |            |. acquire...>|              |
    |<-PyGILState_Ensure--|            |             |              |
    |       ...          ...          ...           ...
    |Stop    #
    |-------`|
    |
    |----Ensure rtns-----># PyObject_  |                            |
             |   :        |CallMethod  |             |              |
             |   :        |.(Doit)...> |. acquire...>| DEADLOCK     |
                 :
                 :
                 :.how does python thread tell
                    PyScheduler not to give away
                      Gil until we are done with
                          critical section??

So my question is how in python do I tell the scheduler not to prempt
the current python thread. Especially how to tell it not to give the
GIL to any calls to PyGILState_ensure until further notice (i.e. until
I am outside my critical section?? It sounds like a reasonable request
- surely this exists already but I can't find it.

One thing that may work (though the documentation does not
specifically say so) is using setcheckinterval() to set the check
interval to a very large value, acessing my shared structure and then
setting the check interval back to the previous value. Provided my
access to the shared structure takes less byte codes than what I set
checkinterval to I should be okay. However that would be very
dependant on the exact fine detail of how the check interval works and
may not be compatible with other Python releases

Maybe someone familiar with the python source code would know for
sure?

I am using Python 2.4.3 on windows XP.

Thanks for any help/suggestions offered!
BR,
Billy.