[Python-ideas] solving multi-core Python

[Nick Coghlan]

On 21 June 2015 at 19:48, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 21 Jun 2015 16:31:33 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> For inter-interpreter communication, the worst case scenario is having
>> to rely on a memcpy based message passing system (which would still be
>> faster than multiprocessing's serialisation + IPC overhead)
>
> And memcpy() updates pointer references to dependent objects magically?
> Surely you meant the memdeepcopy() function that's part of every
> standard C library!

We already have the tools to do deep copies of object trees (although
I'll concede I *was* actually thinking in terms of the classic C/C++
mistake of carelessly copying pointers around when I wrote that
particular message). One of the options for deep copies tends to be a
pickle/unpickle round trip, which will still incur the serialisation
overhead, but not the IPC overhead.

"Faster message passing than multiprocessing" sets the baseline pretty
low, after all.

However, this is also why Eric mentions the notions of object
ownership or limiting channels to less than the full complement of
Python objects. As an *added* feature at the Python level, it's
possible to initially enforce restrictions that don't exist in the C
level subinterpeter API, and then work to relax those restrictions
over time.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia