2.6, 3.0, and truly independent intepreters

[Glenn Linderman]

On approximately 10/30/2008 6:26 AM, came the following characters from 
the keyboard of Jesse Noller:
> On Wed, Oct 29, 2008 at 8:05 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
>   
>> On approximately 10/29/2008 3:45 PM, came the following characters from the
>> keyboard of Patrick Stinson:
>>     
>>> If you are dealing with "lots" of data like in video or sound editing,
>>> you would just keep the data in shared memory and send the reference
>>> over IPC to the worker process. Otherwise, if you marshal and send you
>>> are looking at a temporary doubling of the memory footprint of your
>>> app because the data will be copied, and marshaling overhead.
>>>       
>> Right.  Sounds, and is, easy, if the data is all directly allocated by the
>> application.  But when pieces are allocated by 3rd party libraries, that use
>> the C-runtime allocator directly, then it becomes more difficult to keep
>> everything in shared memory.
>>
>> One _could_ replace the C-runtime allocator, I suppose, but that could have
>> some adverse effects on other code, that doesn't need its data to be in
>> shared memory.  So it is somewhat between a rock and a hard place.
>>
>> By avoiding shared memory, such problems are sidestepped... until you run
>> smack into the GIL.
>>     
>
> If you do not have shared memory: You don't need threads, ergo: You
> don't get penalized by the GIL. Threads are only useful when you need
> to have that requirement of large in-memory data structures shared and
> modified by a pool of workers.

The whole point of this thread is to talk about large in-memory data 
structures that are shared and modified by a pool of workers.

My reference to shared memory was specifically referring to the concept 
of sharing memory between processes... a particular OS feature that is 
called shared memory.

The need for sharing memory among a pool of workers is still the 
premise.  Threads do that automatically, without the need for the OS 
shared memory feature, that brings with it the need for a special 
allocator to allocate memory in the shared memory area vs the rest of 
the address space.

Not to pick on you, particularly, Jesse, but this particular response 
made me finally understand why there has been so much repetition of the 
same issues and positions over and over and over in this thread: instead 
of comprehending the whole issue, people are responding to small 
fragments of it, with opinions that may be perfectly reasonable for that 
fragment, but missing the big picture, or the explanation made when the 
same issue was raised in a different sub-thread.

-- 
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking