Bi-directional sub-process communication

[Israel Brewster]

On Nov 23, 2015, at 1:43 PM, Chris Kaynor <ckaynor at zindagigames.com> wrote:
> 
> On Mon, Nov 23, 2015 at 2:18 PM, Israel Brewster <israel at ravnalaska.net>
> wrote:
> 
>> Of course, that last step could be interesting - implementing the block in
>> such a way as to not tie up the processor, while still getting the data "as
>> soon" as it is available. Unless there is some sort of built-in
>> notification system I could use for that? I.e. the thread would "subscribe"
>> to a notification based on its tag, and then wait for notification. When
>> the master processing thread receives data with said tag, it adds it to the
>> dictionary and "publishes" a notification to that tag. Or perhaps the
>> notification itself could contain the payload?
> 
> 
> There are a few ways I could see handling this, without having the threads
> spinning and consuming CPU:
> 
>   1. Don't worry about having the follow-up code run in the same thread,
>   and use a simple callback. This callback could be dispatched to a thread
>   via a work queue, however you may not get the same thread as the one that
>   made the request. This is probably the most efficient method to use, as the
>   threads can continue doing other work while waiting for a reply, rather
>   than blocking. It does make it harder to maintain state between the pre-
>   and post-request functions, however.
>   2. Have a single, global, event variable that wakes all threads waiting
>   on a reply, each of which then checks to see if the reply is for it, or
>   goes back to sleep. This is good if most of the time, only a few threads
>   will be waiting for a reply, and checking if the correct reply came in is
>   cheap. This is probably good enough, unless you have a LOT of threads
>   (hundreds).
>   3. Have an event per thread. This will use less CPU than the second
>   option, however does require more memory and OS resources, and so will not
>   be viable for huge numbers of threads, though if you hit the limit, you are
>   probably using threads wrong.
>   4. Have an event per request. This is only better than #3 if a single
>   thread may make multiple requests at once, and can do useful work when any
>   of them get a reply back (if they need all, it will make no difference).
> 
> Generally, I would use option #1 or #2. Option 2 has the advantage of
> making it easy to write the functions that use the functionality, while
> option 1 will generally use fewer resources, and allows threads to continue
> to be used while waiting for replies. How much of a benefit that is depends
> on exactly what you are doing.

While I would agree with #1 in general, the threads, in this case, are CherryPy threads, so I need to get the data and return it to the client in the same function call, which of course means the thread needs to block until the data is ready - it can't return and let the result be processed "later".

Essentially there are times that the web client needs some information that only the Child process has. So the web client requests the data from the master process, and the master process then turns around and requests the data from the child, but it needs to get the data back before it can return it to the web client. So it has to block waiting for the data.

Thus we come to option #2 (or 3), which sounds good but I have no clue how to implement :-) Maybe something like http://pubsub.sourceforge.net ? I'll dig into that.

> 
> Option #4 would probably be better implemented using option #1 in all cases
> to avoid problems with running out of OS memory - threading features
> generally require more limited OS resources than memory. Option #3 will
> also often run into the same issues as option #4 in the cases it will
> provide any benefit over option #2.
> 
> Chris
> -- 
> https://mail.python.org/mailman/listinfo/python-list