2.6, 3.0, and truly independent intepreters

[Walter Overby]

Hi,

I've been following this discussion, and although I'm not nearly the
Python expert that others on this thread are, I think I understand
Andy's point of view.  His premises seem to include at least:

1. His Python code does not control the creation of the threads.  That
is done "at the app level".
2. Perhaps more importantly, his Python code does not control the
allocation of the data he needs to operate on.  He's got, for example,
"an opaque OS object" that is manipulated by CPU-intensive OS
functions.

sturlamolden suggests a few approaches:

> 1. Check if a NumPy record array may suffice (dtypes may be nested).
> It will if you don't have dynamically allocated pointers inside the
> data structure.

I suspect that the OS is very likely to have dynamically allocated
pointers inside their opaque structures.

> 2. Consider using multiprocessing's proxy objects or outproc ActiveX
> objects.

I don't understand how this would help.  If these large data
structures reside only in one remote process, then the overhead of
proxying the data into another process for manipulation requires too
much IPC, or at least so Andy stipulates.

> 3. Go to http://pyro.sourceforge.net, download the code and read the
> documentation.

I don't see how this solves the problem with 2.  I admit I have only
cursory knowledge, but I understand "remoting" approaches to have the
same weakness.

I understand Andy's problem to be that he needs to operate on a large
amount of in-process data from several threads, and each thread mixes
CPU-intensive C functions with callbacks to Python utility functions.
He contends that, even though he releases the GIL in the CPU-bound C
functions, the reacquisition of the GIL for the utility functions
causes unacceptable contention slowdowns in the current implementation
of CPython.

After reading Martin's posts, I think I also understand his point of
view.  Is the time spent in these Python callbacks so large compared
to the C functions that you really have to wait?  If so, then Andy has
crossed over into writing performance-critical code in Python.  Andy
proposes that the Python community could work on making that possible,
but Martin cautions that it may be very hard to do so.

If I understand them correctly, none of these concerns are silly.

Walter.