[Python-Dev] Making python C-API thread safe (try 2)

[Jeremy Hylton]

On Thu, 2003-09-11 at 14:50, Shane Hathaway wrote:
> Imagine the following Python module:
> 
> 
> import pseudothreads
> 
> data = pseudothreads.shared([])
> 
> def data_collection_thread():
>      s = get_some_data()
>      data.append(s)
> 
> for n in range(4):
>      pseudothreads.start_new_thread(data_collection_thread)
> 
> 
> In this made-up example, nothing is shared between threads except for 
> the "data" global.  The shared() function copies the list to shared 
> memory and returns a wrapper around the list that prevents access by 
> multiple threads simultaneously.  start_new_thread() is a thin wrapper 
> around os.fork().  Each pseudothread has its own global interpreter lock.
> 
> I wonder whether others would consider such a thing valuable, or even 
> feasible. :-)

This sounds like POSH.  There was a paper about it at PyCon.

http://poshmodule.sourceforge.net/posh/html/

Python uses a single global lock known as the global interpreter lock
(or GIL) to serialize execution of byte codes.  The GIL becomes a
major bottleneck when executing multi-threaded Python applications, in
particular on multi-processor architectures.  This paper presents
POSH, which is an extension module to Python that attempts to address
the problems associated with the GIL by enabling placement of Python
objects in shared memory.  In particular, POSH allows multiple
processes to share objects in much the same way that threads do with
standard Python objects.  We have found that the use of POSH allows
some applications to be structured as if they used threads, but
without the GIL bottleneck.

Jeremy