Execute in a multiprocessing child dynamic code loaded by the parent process

[Martin Di Paola]

>I'm not so sure about that. The author of the plugin knows they're
>writing code that will be dynamically loaded, and can therefore
>expect the kind of problem they're having. It could be argued that
>it's their responsibility to ensure that all the needed code is loaded
>into the subprocess.

Yes but I try to always make my libs/programs as much as usable as
possible. "Ergonomic" would be the word.

In the case of the plugin-engine I'm trying to hide any side-effect or
unexpected behaviour of the engine so the developer of the plugin
does not have take that into account.

I agree that if the developer uses multiprocessing he/she needs to know
its implications. But if I can "smooth" any rough corner, I will try to
do it.

For example, the main project (developed by me) uses threads for
concurrency. It would be simpler to load the plugins and instantiate
them *once* and ask the plugins developers to take care of any
race condition (RC) within their implementation.

Because the plugins were instantiated *once*, it is almost guaranteed
that the plugins will suffer from race conditions and they will require
some sort of locking.

This is quite risky: you may forget to protect something and you will
end up with a RC and/or you may put the lock in the wrong place and the
whole thing will not work concurrently.

My decision back then was to instantiate each plugin N+1 times: once in
the main thread and then once per worker thread.

With this, no single plugin instance will be shared so there is no risk
of RC and no need for locking. (Yes, I know, the developer just needs to
use a module variable or a class attribute and it will get a RC and
these are shared but it is definitely not the default scenario).

If sharing is required I provide an object that minimizes the locking
needed.

It was much complex for me at the design and at the implementation level
but I think that it is safer and requires less from the plugin
developer.

Reference: https://byexamples.github.io/byexample/contrib/concurrency-model