Execute in a multiprocessing child dynamic code loaded by the parent process

[Chris Angelico]

On Mon, 7 Mar 2022 at 07:37, Martin Di Paola <martinp.dipaola at gmail.com> wrote:
>
>
>
> >
> >The way you've described it, it's a hack. Allow me to slightly redescribe it.
> >
> >modules = loader()
> >objs = init(modules)
> >
> >def invoke(mod, func):
> >    # I'm assuming that the loader is smart enough to not load
> >    # a module that's already loaded. Alternatively, load just the
> >    # module you need, if that's a possibility.
> >    loader()
> >    target = getattr(modules[mod], func)
> >    target()
> >
> >ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi"))
> >ch.start()
> >
>
> Yeup, that would be my first choice but the catch is that "sayhi" may
> not be a function of the given module. It could be a static method of
> some class or any other callable.

Ah, fair. Are you able to define it by a "path", where each step in
the path is a getattr() call?

The trouble is, arbitrary callables might not be available in a
reconstructed version of the module.

> Using multiprocessing.reduction was a practical decision: if the user
> wants to call something non-pickleable, it is not my fault, it is
> multiprocessing's fault.
>
> It *would* be my fault if multiprocessing.Process fails only because I'm
> loading the code dynamically.

Fair. I guess, then, that the best thing to do is to preload the
modules, then unpickle. So, basically what you already have, but with
more caveats.

> Do you have some in mind? Or may be a project that I could read?

Not handy, but there are always many different ways to do things. For
instance, instead of saying "spawn a subprocess and call this
function", you could invert it, and have the function register itself
as the target. Then it's just "spawn a subprocess and load this
module", and that calls the registered invocation. It all depends on
what the rest of your project is doing. Mainly, though, I'm just not
ruling out the possibility of other options :)

ChrisA