Persistent variable in subprocess using multiprocessing?

[mheavner]

Piet,

The situation is 1a of your listed options, however my issue was
solved. I was stopping the subprocesses from consuming more data at
each iteration which led to the data being lost since the subprocess
worker function would then end - I now keep them alive across
iterations.

Thanks for your help, I'm new to the multiprocessing module and this
was very helpful!

On Jul 17, 4:26 am, Piet van Oostrum <p... at cs.uu.nl> wrote:
> There is stil something not clear in your description.
>
> >m> I'm using multiprocessing to spawn several subprocesses, each of which
> >m> uses a very large data structure (making it impractical to pass it via
> >m> pipes / pickling). I need to allocate this structure once when the
> >m> process is created and have it remain in memory for the duration of
> >m> the process.
>
> I have read this as that every subprocess has its own large
> data structure and that there is no connection between these.
>
> But seeing where the discussion is going I guess there might be
> different interpretations. So can you enlighten us how the situation is?
>
> 1. Each subprocess has a copy of a data structure that is prepared by the
>    master process. Therefore you want it to be passed by the fork
>    1a. the data structure is constant i.e. the subprocess doesn't change it
>    1b. the subprocess makes changes in its copy
> 2. Each subprocess has a seperate data structure not equal to the others
> 3. Something else.
> --
> Piet van Oostrum <p... at cs.uu.nl>
> URL:http://pietvanoostrum.com[PGP 8DAE142BE17999C4]
> Private email: p... at vanoostrum.org