multiprocessing eats memory

[redbaron]

On 26 сент, 17:03, MRAB <goo... at mrabarnett.plus.com> wrote:
> On Sep 26, 9:52 am, redbaron <ivanov.ma... at gmail.com> wrote:
>
> > On 26 ÓÅÎÔ, 04:20, Istvan Albert <istvan.alb... at gmail.com> wrote:
>
> > > On Sep 25, 8:40šam, "Max Ivanov" <ivanov.ma... at gmail.com> wrote:
>
> > > > At any time in main process there are shouldn't be no more than two copies of data
> > > > (one original data and one result).
>
> > > From the looks of it you are storing a lots of references to various
> > > copies of your data via the async set.
>
> > How could I avoid of storing them? I need something to check does it
> > ready or not and retrieve results if ready. I couldn't see the way to
> > achieve same result without storing asyncs set.
>
> You could give each worker process an ID and then have them put the ID
> into a queue to signal to the main process when finished.
And how could I retrieve result from worker process without async?

>
> BTW, your test-case modifies the asyncs set while iterating over it,
> which is a bad idea.
My fault, there was list(asyncs) originally.