[Python-ideas] solving multi-core Python

[M.-A. Lemburg]

On 24.06.2015 10:22, Nick Coghlan wrote:
> On 24 June 2015 at 15:33, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> On Sun, Jun 21, 2015 at 7:06 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Nick Coghlan schrieb am 21.06.2015 um 03:28:
>>>> * there may be restrictions on some extension modules that limit them
>>>> to "main interpreter only" (e.g. if the extension module itself isn't
>>>> thread-safe, then it will need to remain fully protected by the GIL)
>>>
>>> Just an idea, but C extensions could opt-in to this. Calling into them has
>>> to go through some kind of callable type, usually PyCFunction. We could
>>> protect all calls to extension types and C functions with a global runtime
>>> lock (per process, not per interpreter) and Extensions could set a flag on
>>> their functions and methods (or get it inherited from their extension types
>>> etc.) that says "I don't need the lock". That allows for a very
>>> fine-grained transition.
>>
>> Exactly.  PEP 489 helps facilitate opting in as well, right?
> 
> Yep, as PEP 489 requires subinterpreter compatibility as a
> precondition for using multi-phase initialisation :)
> 
> Cheers,
> Nick.
> 
> P.S. Technically, what it actually requires is support for "multiple
> instances of the module existing in the same process at the same
> time", as it really recreates the module if you remove it from
> sys.modules and import it again, unlike single phase initialisation.
> But that's a mouthful, so "must support subinterpreters" is an easier
> shorthand.

Note that extension modules often interface to other C libraries
which typically use some setup logic that is not thread safe,
but is used to initialize the other thread safe parts. E.g.
setting up locks and shared memory for all threads to
use is a typical scenario you find in such libs.

A requirement to be able to import modules multiple times
would pretty much kill the idea for those modules.

That said, I don't think this is really needed. Modules
would only have to be made aware that there is a global
first time setup phase and a later shutdown/reinit phase.

As a result, the module DLL would load only once, but then
use the new module setup logic to initialize its own state
multiple times.

That said, I still think the multiple-process is a better one (more
robust, more compatible, fewer problems). We'd just need a way more
efficient approach to sharing objects between the Python processes
than using pickle and shared memory or pipes :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 24 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78
2015-07-20: EuroPython 2015, Bilbao, Spain ...             26 days to go
2015-07-29: Python Meeting Duesseldorf ...                 35 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/