[Python-ideas] solving multi-core Python

[Steven D'Aprano]

On Sat, Jun 20, 2015 at 03:42:33PM -0600, Eric Snow wrote:

> * only allow passing plain functions to Task() and
> Subinterpreter.run() (exclude closures, other callables)

That doesn't sound very Pythonic to me. That's going to limit the 
usefulness of these subinterpreters.


> * object ownership model
>   + read-only in all but 1 subinterpreter
>   + RW in all subinterpreters

Isn't that a contradiction? If objects are read-only in all 
subinterpreters (except one), how can they be read/write in all 
subinterpreters?


All this talk about subinterpreters reminds me of an interesting blog 
post by Armin Ronacher:

http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see

He's quite critical of a number of internal details of the CPython 
interpreter. But what I take from his post is that there could be 
significant advantages to giving the CPython interpreter its own local 
environment, like Lua and Javascript typically do, rather than the 
current model where there is a single process-wide global environment. 
Instead of having multiple subinterpreters all running inside the main 
interpreter, you could have multiple interpreters running in the same 
process, each with their own environment.

I may be completely misinterpreting things here, but as I understand it, 
this would remove the need for the GIL, allowing even plain old threads 
to take advantage of multiple cores. But that's a separate issue.

Armin writes:

    I would like to see an internal interpreter design could be based on 
    interpreters that work independent of each other, with local base 
    types and more, similar to how JavaScript works. This would 
    immediately open up the door again for embedding and concurrency 
    based on message passing. CPUs won't get any faster :)

(He also talks about CPython's tp_slots system, but that's a 
separate issue, I think.)

Now I have no idea if Armin is correct, or whether I am even 
interpreting his post correctly. But I'd like to hear people's thoughts 
on how this might interact with Eric's suggestion.


-- 
Steve