[C++-sig] Cplusplus-sig Digest, Vol 17, Issue 4

[Pascal Briet]

Thanks to Renato, the solution was just... PyEval_InitThreads ! 
Thanks for your help !


Matthew, you're right, there is just one thread running at a time in Python.

But, if my understanding of the Python API is correct, more than one thread
can be registered to the Python Interpreter (PyThreadState). Calling
PyGILState_Ensure function creates a PyThreadState.
Next, the Python API will switch its current thread every 100 bytecodes
(this value is the default of sys.setcheckinterval function). There are
never two system threads running at the same time in the Python API, but at
least... they are alternated.


One proof is that PyGILState_Ensure is _not_ blocking the second thread
during the execution of the first thread in Python. It probably waits a few
microseconds, as I explain below, but the 2 threads enter the "critical"
Python part together. I checked that :

static int count = 0;

 
_gstate = PyGILState_Ensure();
if (count != 0)
	// In my code, I use a thread-safe stream.
	std::cout << "2 threads together" << std::endl; 
count++;

 // Interprets Python bytecodes with boost::python

count--;
PyGILState_Release(_gstate);


The message "2 threads together" appears. Without PyEval_InitThreads, it
crashes just after.


You are right about the locks, they are not needed.

Let's detail what happens :

a) Thread 1 : GIL_Ensure(New PyThread_State registered, GIL lock)
b) Thread 1 : Executing Python through the Interpreter   |   Thread 2 :
GIL_Ensure (waiting for GIL)
c) Thread 1 : Executing the 100th bytecode : Python Interpreter Releases the
GIL. Switch to Thread 2.
d) Thread 1 : waiting for GIL  | Thread 2 : Executing Python
e) Thread 2 : 100th bytecode : Switch to Thread 1.
f) Thread 1 : Executing Python | Thread 2 : Waiting for GIL
etc.


-- 
BRIET Pascal


------------------------------

Remember of use PyEval_InitThreads on your module initialization.

Renato Araujo Oliveira Filho


------------------------------

Message: 3
Date: Tue, 2 Feb 2010 11:30:37 -0600
From: "Matthew Scouten (TT)" <Matthew.Scouten at tradingtechnologies.com>
To: "Development of Python/C++ integration" <cplusplus-sig at python.org>
Subject: Re: [C++-sig] PyGILState_Release with multithread
Message-ID: <32490DFF7774554A85D65D23A9F0F9380C4DEE5F at chiex01>
Content-Type: text/plain; charset="iso-8859-1"

First of all, the PyGILState_* functions are acquiring and releasing the
"Global Interpreter _Lock_" Your _pyMutex is redundant. 

Here are the rules that govern the GIL:

1)      Only one thread at a time can hold the GIL

2)      Any thread that is touching python data, python code, or any part of
the 'terp must hold the GIL

 

The python part of your program WILL be single threaded. There is nothing
you can do about that. 

 

From:
cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org
[mailto:cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python
.org] On Behalf Of Pascal Briet
Sent: Tuesday, February 02, 2010 5:46 AM
To: cplusplus-sig at python.org
Subject: [C++-sig] PyGILState_Release with multithread

 

Hello,

 

After a few days of headache, I think the best way is to share this
problematic...

 

I have :

- a main Python script

- a C module imported and called from Python  (thanks to boost ::python)

- A dozen of threads created by a C function  (boost ::thread)

- These threads, in a C main loop, call regularly a Python function.

 

 

Here is the critical part of the C code, with the Python GIL management :

 

boost::mutex      _pyMutex;

PyGILState_STATE  _gstate;

 

_pyMutex.lock();

_gstate = PyGILState_Ensure();

_pyMutex.unlock();

 

// Interprets Python bytecodes with boost::python

 

_pyMutex.lock();

PyGILState_Release(_gstate);

_pyMutex.unlock();

 

 

This part is multi-threaded, and it seems that the PyGILState_* functions do
not like it.

I have the following error with PyGILState_Release : ? This thread state
must be current when releasing ?

 

I think that when 2 threads enter PyGILState_Ensure, the second one is
considered as the current one. When the first one end...  it is not current.
A quick look to the code of the Python API seems to confirm it.

 

 

So, the only solution I found is to lock the whole call to Python :

 

boost::mutex      _pyMutex; 

PyGILState_STATE  _gstate;

 

_pyMutex.lock();

_gstate = PyGILState_Ensure();

 

// Interprets Python bytecodes with boost::python

 

PyGILState_Release(_gstate);

_pyMutex.unlock();

 

 

Everything is ok with this solution, except that...  only one thread at a
time can call a Python method.

When 2 threads ask for a Python call, if the first one is a blocking
operation, the second one will wait...  it is so bad !

 

Do you have any solution ?

 

Nb : I use Python 2.5.2

 

Thanks for your help.