2.6, 3.0, and truly independent intepreters

"Martin v. Löwis" martin at v.loewis.de
Fri Oct 24 21:52:58 EDT 2008


>> A c-level module, on the other hand, can sidestep/release
>> the GIL at will, and go on it's merry way and process away.
> 
> ...Unless part of the C module execution involves the need do CPU-
> bound work on another thread through a different python interpreter,
> right? 

Wrong.

> (even if the interpreter is 100% independent, yikes).

Again, wrong.

> For
> example, have a python C module designed to programmatically generate
> images (and video frames) in RAM for immediate and subsequent use in
> animation.  Meanwhile, we'd like to have a pthread with its own
> interpreter with an instance of this module and have it dequeue jobs
> as they come in (in fact, there'd be one of these threads for each
> excess core present on the machine).

I don't understand how this example involves multiple threads. You
mention a single thread (running the module), and you mention designing
a  module. Where is the second thread?

Let's assume there is another thread producing jobs, and then
a thread that generates the images. The structure would be this

  while 1:
    job = queue.get()
    processing_module.process(job)

and in process:

  PyArg_ParseTuple(args, "s", job_data);
  result = PyString_New(bufsize);
  buf = PyString_AsString(result);
  Py_BEGIN_ALLOW_THREADS
    compute_frame(job_data, buf);
  Py_END_ALLOW_THREADS
  return PyString_FromString(buf);

All these compute_frames could happily run in parallel.

> As far as I can tell, it seems
> CPython's current state can't CPU bound parallelization in the same
> address space.

That's not true.

Regards,
Martin



More information about the Python-list mailing list