parallel processing in standard library

[Robert Kern]

Emin.shopper Martinian.shopper wrote:
> Dear Experts,
> 
> Is there any hope of a parallel processing toolkit being incorporated
> into the python standard library? I've seen a wide variety of toolkits
> each with various features and limitations. Unfortunately, each has its
> own API. For coarse-grained parallelism, I suspect I'd be pretty happy
> with many of the existing toolkits, but if I'm going to pick one API to
> learn and program to, I'd rather pick one that I'm confident is going to
> be supported for a while.
> 
> So is there any hope of adoption of a parallel processing system into
> the python standard library? If not, is there any hope of something like
> the db-api for coarse grained parallelism (i.e, a common API that
> different toolkits can support)?

The problem is that for SQL databases, there is a substantial API that they can
all share. The implementations are primarily differentiated by other factors
like speed, in-memory or on-disk, embedded or server, the flavor of SQL, etc.
and only secondarily differentiated by their extensions to the DB-API. With
parallel processing, the API itself is a key differentiator between toolkits and
approaches. Different problems require different APIs, not just different
implementations.

I suspect that one of the smaller implementations like processing.py might get
adopted into the standard library if the author decides to push for it. The ones
I am thinking of are relatively new, so I imagine that it might take a couple of
years of vigorous use by the community before it gets into the standard library.

My recommendation to you is to pick one of the smaller implementations that
solves the problems in front of you. Read and understand that module so you
could maintain it yourself if you had to. Post to this list about how you use
it. Blog about it if you blog. Write some Python Cookbook recipes to show how
you solve problems with it. If there is a lively community around it, that will
help it get into the standard library. Things get into the standard library
*because* they are supported, not the other way around.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco