Bi-directional sub-process communication

[Cameron Simpson]

On 23Nov2015 14:14, Israel Brewster <israel at ravnalaska.net> wrote:
>On Nov 23, 2015, at 1:43 PM, Chris Kaynor <ckaynor at zindagigames.com> wrote:
>> On Mon, Nov 23, 2015 at 2:18 PM, Israel Brewster <israel at ravnalaska.net>
>> wrote:
>>   3. Have an event per thread. This will use less CPU than the second
>>   option, however does require more memory and OS resources, and so will not
>>   be viable for huge numbers of threads, though if you hit the limit, you are
>>   probably using threads wrong.
[...]
>While I would agree with #1 in general, the threads, in this case, are 
>CherryPy threads, so I need to get the data and return it to the client in the 
>same function call, which of course means the thread needs to block until the 
>data is ready - it can't return and let the result be processed "later".

Then #3. I would have a common function/method for submitting a request to go 
to the subprocess, and have that method return an Event on which to wait. Then 
caller then just waits for the Event and collects the data. Obviously, the 
method does not just return the Event, but an Event and something to receive 
the return data. I've got a class called a Result for this kind of thing; make 
a small class containing an Event and which will have a .result attribute for 
the return information; the submitting method allocates one of these and 
returns it. The response handler gets the instance (by looking it up from the 
tag), sets the .result attribute and fires the Event. Your caller wakes up from 
waiting on the Event and consults the .result attribute.

Completely untested example code:

  class ReturnEvent:
    def __init__(self):
      self.event = Event()

  seq = 0
  re_by_tag = {}

  def submit_request(query):
    global seq, re_by_tag
    tag = seq
    seq += 1
    RE = ReturnEvent()
    re_by_tag[tag] = RE
    send_request(tag, query)
    return RE

  def process_response(tag, response_data):
    RE = re_by_tag.pop(tag)
    RE.result = response_data
    RE.event.set()

  ... CherryPy request handler ...
  RE = submit_request(your_query_info)
  RE.wait()
  response_data = RE.result

Cheers,
Cameron Simpson <cs at zip.com.au>