[Python-ideas] solving multi-core Python

[Nick Coghlan]

On 24 June 2015 at 21:43, M.-A. Lemburg <mal at egenix.com> wrote:
>
> Note that extension modules often interface to other C libraries
> which typically use some setup logic that is not thread safe,
> but is used to initialize the other thread safe parts. E.g.
> setting up locks and shared memory for all threads to
> use is a typical scenario you find in such libs.
>
> A requirement to be able to import modules multiple times
> would pretty much kill the idea for those modules.

Yep, that's the reason earlier versions of PEP 489 included the notion
of "singleton modules". We ended up deciding to back that out for the
time being, and instead leave those modules using the existing single
phase initialisation model.

> That said, I don't think this is really needed. Modules
> would only have to be made aware that there is a global
> first time setup phase and a later shutdown/reinit phase.
>
> As a result, the module DLL would load only once, but then
> use the new module setup logic to initialize its own state
> multiple times.

Aye, buying more time to consider alternative designs was the reason
we dropped the "singleton module" idea from multi-phase initialisation
until 3.6 at the earliest. I think your idea here has potential - it
should just require a new Py_mod_setup slot identifier, and a bit of
additional record keeping to track which modules had already had their
setup slots invoked. (It's conceivable there could also be a
process-wide Py_mod_teardown slot, but that gets messy in the embedded
interpreter case where we might have multiple
Py_Initialize/Py_Finalize cycles)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia