Bi-directional sub-process communication

[Israel Brewster]

On Nov 23, 2015, at 12:45 PM, Cameron Simpson <cs at zip.com.au> wrote:
> 
> On 23Nov2015 12:22, Israel Brewster <israel at ravnalaska.net> wrote:
>> On Nov 23, 2015, at 11:51 AM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
>>> Concurrency, ugh.
> 
> I'm a big concurrency fan myself.
> 
>>> It's probably better just to have a Condition/Event per thread and
>>> have the response thread identify the correct one to notify, rather
>>> than just notify a single shared Condition and hope the threads wake
>>> up in the right order.
>> 
>> Tell me about it :-) I've actually never worked with conditions or notifications (actually even this bi-drectional type of communication is new to me), so I'll have to look into that and figure it out. Thanks for the information!
> 
> I include a tag with every request, and have the responses include the tag; the request submission function records the response hander in a mapping by tag and the response handing thread looks up the mapping and passes the response to the right handler.
> 
> Works just fine and avoids all the worrying about ordering etc.
> 
> Israel, do you have control over the protocol between you and your subprocess?  If so, adding tags is easy and effective.

I do, and the basic concept makes sense. The one difficulty I am seeing is getting back to the thread that requested the data.  Let me know if this makes sense or I am thinking about it wrong:

- When a thread requests some data, it sends the request as a dictionary containing a tag (unique to the thread) as well as the request
- When the child processes the request, it encodes the response as a dictionary containing the tag and the response data
- A single, separate thread on the "master" side parses out responses as they come in and puts them into a dictionary keyed by tag
- The requesting threads, after putting the request into the Queue, would then block waiting for data to appear under their key in the dictionary

Of course, that last step could be interesting - implementing the block in such a way as to not tie up the processor, while still getting the data "as soon" as it is available. Unless there is some sort of built-in notification system I could use for that? I.e. the thread would "subscribe" to a notification based on its tag, and then wait for notification. When the master processing thread receives data with said tag, it adds it to the dictionary and "publishes" a notification to that tag. Or perhaps the notification itself could contain the payload? 

Thanks for the information!

> 
> Cheers,
> Cameron Simpson <cs at zip.com.au>