Share unpickleable object across processes

[Chris Angelico]

On Fri, Nov 3, 2017 at 11:00 AM, Dennis Lee Bieber
<wlfraed at ix.netcom.com> wrote:
> On Thu, 2 Nov 2017 12:32:35 -0800, Israel Brewster <israel at ravnalaska.net>
> declaimed the following:
>
>
>>
>>I *must* be thinking about this wrong. Take even a basic chat app that uses websockets. Client a, which connected to process 1, sends a message to the server. There are three other clients connected, each of which needs to receive said message. Given that the way UWSGI works said clients could have connected to any one of the worker processes, how can the server push the message out to *all* clients? What am I missing here?
>>
>
>         This is beginning to sound like a form of publisher/subscriber system.
>
> http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pubsub.html
>
>         Though I have to admit I don't quite see how to set up bidirectional
> communication. Having the workers subscribe to receive data from a single
> publisher is the easier part -- but how a worker can send data to the
> publisher for distribution is not clear; the distribution process can't
> subscribe to every worker automatically... Perhaps something using
>
> http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/devices/forwarder.html
>
> with each worker making both publisher and subscriber connections. That
> could let a worker publish to the forwarder, which then distributes to all
> subscribed workers. Might need some way to identify one's own message
> (perhaps each worker has a topic, and filters its own topic out of the
> return stream).

Websockets are perfect for this sort of thing. It's not difficult....
but you need to have all the clients managed by a single central hub,
which is the exact problem the OP faced.

Here's a simple websocket app that does this sort of thing:

https://github.com/Rosuav/monopoly-open-bid/blob/master/server.py

It depends on running a single process to handle every client, and
then simply maintains an in-memory list of concurrent clients. (This
uses asyncio, but equivalent models using threads are also possible.)

ChrisA