Manipulate GIL to have Python thread work with pthread native ones

[qshhnkf at alumni.sjtu.edu.cn]

Hi Python experts, 

I need to extend Python with C/C++ to interact with an in-house legacy API. I'm implementing a data subscriber with Python, which subscribes to a data publisher (actually a ZeroMQ publisher socket) and will get notified once any new messages are fed. In my subscriber, the messages are dumped to a data processor after received. The subscriber will also get notified by the processor when done. Since the data processor is written in C++, I have to extend the Python code with a simple C++ module. 



Below is a simplified runnable code sample of my data subscriber. Code main.py , in which the module proc represents the processor, subscribe to a ZeroMQ socket on localhost:10000 , setup the callback, and send the received message to the processor by calling proc.onMsg . 


#!/bin/python
# main.py

import gevent
import logging
import zmq.green as zmq

import pub 
import proc

logging.basicConfig( format='[%(levelname)s] %(message)s', level=logging.DEBUG )

SUB_ADDR = 'tcp://localhost:10000'

def setupMqAndReceive():
    '''Setup the message queue and receive messages.
    '''
    ctx  = zmq.Context()
    sock = ctx.socket( zmq.SUB )
    # add topics
    sock.setsockopt_string( zmq.SUBSCRIBE, 'Hello' )

    sock.connect( SUB_ADDR )

    while True:
        msg = sock.recv().decode( 'utf-8' )
        proc.onMsg( msg )

def callback( a, b ):
    print( '[callback]',  a, b ) 

def main():
    '''Entrance of the module.
    '''
    pub.start()
    proc.setCallback( callback )
    '''A simple on-liner
    gevent.spawn( setupMqAndReceive ).join()
    works. However, the received messages will not be
    processed by the processor.
    '''
    gevent.spawn( setupMqAndReceive )
    proc.start() 


Module proc is simplified with three functions exported: 

    * setCallback setup the callback function, so that when the message processed, my subscriber could be notified; 
    * onMsg is invoked by the subscriber; 
    * start setups a new worker thread to handle the messages from subscriber and make the main thread join to wait for the worker thread to exit. 


The full version of source code could be found on github at https://github.com/more-more-tea/python_gil . Nevertheless, it does not run as my expectation. Once the processor thread is added, the subscriber cannot receive data from the publisher in the gevent loop. If I simply drop the data processor module, the subscriber gevent loop could receive the messages from the publisher. The same behavior could be observed with threading.Thread instead of gevent. 




I posted the same question on StackOverflow 

http://stackoverflow.com/questions/37513957/does-python-gil-need-to-be-taken-care-when-work-with-multi-thread-c-extension 

with one answer direct me to the article 

https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock 

I read the document but still not sure where to add the 

PyGILState_Ensure/PyGILState_Release. Without a running sample, it is a little bit hard to understand how Python thread and the native pthread interact. 

Would you mind review the source code and direct the right way to make it running? 

Thank you very much for your time and best regards! 
Shuang