multiprocessing eats memory

[MRAB]

On Sep 26, 9:52 am, redbaron <ivanov.ma... at gmail.com> wrote:
> On 26 ÓÅÎÔ, 04:20, Istvan Albert <istvan.alb... at gmail.com> wrote:
>
> > On Sep 25, 8:40šam, "Max Ivanov" <ivanov.ma... at gmail.com> wrote:
>
> > > At any time in main process there are shouldn't be no more than two copies of data
> > > (one original data and one result).
>
> > From the looks of it you are storing a lots of references to various
> > copies of your data via the async set.
>
> How could I avoid of storing them? I need something to check does it
> ready or not and retrieve results if ready. I couldn't see the way to
> achieve same result without storing asyncs set.

You could give each worker process an ID and then have them put the ID
into a queue to signal to the main process when finished.

BTW, your test-case modifies the asyncs set while iterating over it,
which is a bad idea.