Share unpickleable object across processes

[Israel Brewster]

On Nov 2, 2017, at 11:15 AM, Stefan Ram <ram at zedat.fu-berlin.de> wrote:
> 
> Israel Brewster <israel at ravnalaska.net> writes:
>> the data structure that stores the socket connections needs
>> to be shared across all said processes.
> 
>  IIRC that's the difference between threads and
>  processes: threads share a common memory.
> 
>  You can use the standard module mmap to share
>  data between processes.
> 
>  If it's not pickleable, but if you can write code
>  to serialize it to a text format yourself, you
>  can share that text representation via, er, sockets.

If I could serialize it to a text format, then I could pickle said text format and store it in redis/some other third party store. :-)

> 
>> In C I might do something like store a void pointer to the
>> object, then cast it to the correct object type
> 
>  Restrictions of the OS or MMU even apply to
>  C code.

Sure, I was just talking in general "ideas". I'm not saying I tried it or it would work.

> 
>> , but that's not an option in python. So how can I get around
>> this issue?
> 
>  You can always write parts of a CPython program
>  in C, for example, using Cython.

True, but I just need to be able to share this piece of data - I don't want to reinvent the wheel just to write an app that uses web sockets!

I *must* be thinking about this wrong. Take even a basic chat app that uses websockets. Client a, which connected to process 1, sends a message to the server. There are three other clients connected, each of which needs to receive said message. Given that the way UWSGI works said clients could have connected to any one of the worker processes, how can the server push the message out to *all* clients? What am I missing here?

> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list