[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

[Michael Selik]

On Fri, Sep 28, 2018 at 2:11 PM Sean Harrington <seanharr11 at gmail.com> wrote:
> kwarg on Pool.__init__ called `expect_initret`, that defaults to False. When set to True:
> Capture the return value of the initializer kwarg of Pool
> Pass this value to the function being applied, as a kwarg.

The parameter name you chose, "initret" is awkward, because nowhere
else in Python does an initializer return a value. Initializers mutate
an encapsulated scope. For a class __init__, that scope is an
instance's attributes. For a subprocess managed by Pool, that
encapsulated scope is its "globals". I'm using quotes to emphasize
that these "globals" aren't shared.


On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington <seanharr11 at gmail.com> wrote:
> On Fri, Sep 28, 2018 at 6:45 PM Antoine Pitrou <solipsis at pitrou.net> wrote:
>> 3. If you don't like globals, you could probably do something like
>> lazily-initialize the resource when a function needing it is executed
>
> if initializing the resource is expensive, we only want to do this ONE time per worker process.

We must have a different concept of "lazily-initialize". I understood
Antoine's suggestion to be a one-time initialize per worker process.


On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington <seanharr11 at gmail.com> wrote:
> My simple argument is that the developer should not be constrained to make the objects passed globally available in the process, as this MAY break encapsulation for large projects.

I could imagine someone switching from Pool to ThreadPool and getting
into trouble, but in my mind using threads is caveat emptor. Are you
worried about breaking encapsulation in a different scenario?