[Python-ideas] Parallel processing with Python

[Guido van Rossum]

On Thu, Feb 19, 2009 at 2:53 AM, Sturla Molden <sturla at molden.no> wrote:
> On 2/19/2009 3:34 AM, Adam Olsen wrote:
>> * Any communication requires a serialize/copy/deserialize sequence.
>
> No it does not, and this why embedded interpreters are better than multiple
> processes (cf. multiprocessing). Since the interpreters share virtual
> memory, many objects can be shared without any serialization.
> That is, C pointers will be valid in both interpreters, so it should in many
> cases be possible to pass a PyObject* from one interpreter to another. This
> kind of communication would be easiest to achieve with immutable objects.

Only if you have an approach to GC that does not require locking. The
current reference counting macros are not thread-safe so every INCREF
and DECREF would have to be somehow protected by a lock or turned into
an atomic operation. Recall that many frequently used objects, like
None, small integers, and interned strings for common identifiers, are
shared and constantly INCREFed and DECREFed.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)