2.6, 3.0, and truly independent intepreters

[Patrick Stinson]

Speaking of the big picture, is this how it normally works when
someone says "Here's some code and a problem and I'm willing to pay
for a solution?" I've never really walked that path with a project of
this complexity (I guess it's the backwards-compatibility that makes
it confusing), but is this problem just too complex so we have to keep
talking and talking on forum after forum? Afraid to fork? I know I am.
How many people are qualified to tackle Andy's problem? Are all of
them busy or uninterested? Is the current code in a tight spot where
it just can't be fixed without really jabbing that FORK in so deep
that the patch will die when your project does?

Personally I think this problem is super-awesome on the hobbyest's fun
scale. I'd totally take the time to let my patch do the talking but I
haven't read enough of the (2.5) code. So, I resort to simply reading
the newsgroups and python code to better understand the mechanics
problem :(

On Thu, Oct 30, 2008 at 2:54 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
> On approximately 10/30/2008 6:26 AM, came the following characters from the
> keyboard of Jesse Noller:
>>
>> On Wed, Oct 29, 2008 at 8:05 PM, Glenn Linderman <v+python at g.nevcal.com>
>> wrote:
>>
>>>
>>> On approximately 10/29/2008 3:45 PM, came the following characters from
>>> the
>>> keyboard of Patrick Stinson:
>>>
>>>>
>>>> If you are dealing with "lots" of data like in video or sound editing,
>>>> you would just keep the data in shared memory and send the reference
>>>> over IPC to the worker process. Otherwise, if you marshal and send you
>>>> are looking at a temporary doubling of the memory footprint of your
>>>> app because the data will be copied, and marshaling overhead.
>>>>
>>>
>>> Right.  Sounds, and is, easy, if the data is all directly allocated by
>>> the
>>> application.  But when pieces are allocated by 3rd party libraries, that
>>> use
>>> the C-runtime allocator directly, then it becomes more difficult to keep
>>> everything in shared memory.
>>>
>>> One _could_ replace the C-runtime allocator, I suppose, but that could
>>> have
>>> some adverse effects on other code, that doesn't need its data to be in
>>> shared memory.  So it is somewhat between a rock and a hard place.
>>>
>>> By avoiding shared memory, such problems are sidestepped... until you run
>>> smack into the GIL.
>>>
>>
>> If you do not have shared memory: You don't need threads, ergo: You
>> don't get penalized by the GIL. Threads are only useful when you need
>> to have that requirement of large in-memory data structures shared and
>> modified by a pool of workers.
>
> The whole point of this thread is to talk about large in-memory data
> structures that are shared and modified by a pool of workers.
>
> My reference to shared memory was specifically referring to the concept of
> sharing memory between processes... a particular OS feature that is called
> shared memory.
>
> The need for sharing memory among a pool of workers is still the premise.
>  Threads do that automatically, without the need for the OS shared memory
> feature, that brings with it the need for a special allocator to allocate
> memory in the shared memory area vs the rest of the address space.
>
> Not to pick on you, particularly, Jesse, but this particular response made
> me finally understand why there has been so much repetition of the same
> issues and positions over and over and over in this thread: instead of
> comprehending the whole issue, people are responding to small fragments of
> it, with opinions that may be perfectly reasonable for that fragment, but
> missing the big picture, or the explanation made when the same issue was
> raised in a different sub-thread.
>
> --
> Glenn -- http://nevcal.com/
> ===========================
> A protocol is complete when there is nothing left to remove.
> -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
>
>