[concurrency] Issues with the current concurrency mechanisms in Python

[Mike Meyer]

General:

All concurrency is built on top of the OS primitives, with little if
any higher-level constructs hiding the details of managing
concurrency. The exceptions are the queue modules. We need better
tools.

Mutating unprotected shared memory (always an error) passes silently,
which is unpythonic.


Processes:

There's no portable way to get named shared memory.

There's no way to get general python objects into shared memory.

Reference counting causes COW pages to be copied unnecessarily.


Threads:

Most of the standard library has not been audited for thread safety.

Everything is shared by default. Sharing should be explicit, because
"Explicit is better than implicit."

The GIL. Once the GIL is gone, reference counting will slow things
down.


Distributed systems:

[This is basically processes restricted to network communications methods.]

The Processes tools are sufficient for distributed systems, except
they can't be used without the addition of a rendezvous mechanism.


Some proposed solutions:

Add shm_open and shm_unlink to the mmap module. This fixes the issue
of named shared memory for processes.

Add a 'lock' option to mmap.mmap, which will create a lock attribute
for the mmap object with the same API as the threading.Lock/RLock
objects. This is a stopgap for putting python objects in shared
memory, but fills a critical need.

Auditing the standard library, not having unprotected mutations of
shared objects pass silently, and having everything shared by default
are related issues. The latter two require the language system to know
whether an object might be exposed to concurrent access, as the answer
being "no" might allow eliding the handling of it. A tool for
figuring such things out would also be useful as a first step in
auditing the standard library.

    <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org