parallel processing in standard library

[Robert Kern]

Emin.shopper Martinian.shopper wrote:
> On Dec 27, 2007 4:13 PM, Robert Kern <robert.kern at gmail.com
> <mailto:robert.kern at gmail.com>> wrote:

>     My recommendation to you is to pick one of the smaller
>     implementations that
>     solves the problems in front of you. Read and understand that module
>     so you
>     could maintain it yourself if you had to. Post to this list about
>     how you use
>     it. Blog about it if you blog. Write some Python Cookbook recipes to
>     show how
>     you solve problems with it.
> 
> 
> That is a good suggestion, but for most of the coarse grained
> parallelism tasks I've worked on it would be easier to roll my own
> system than do that. To put it another way, why spend the effort to use
> a particular API if I don't know its going to be around for a while?
> Since a lot of the value is in the API as opposed to the implementation,
> unless there is something special about the API ( e.g., it is an
> official or at least de factor standard) the learning curve may not be
> worth it.

And you think that you will encounter no learning curve writing your own code?
At least take the opportunity to see how other people have solved your problem.
Some of the implementations floating around now fit into one module. Surely, it
would take less time to understand one of them than write your own. And let's
not forget testing your module. The initial writing is never the timesink; it's
the testing!

>     If there is a lively community around it, that will
>     help it get into the standard library. Things get into the standard
>     library
>     *because* they are supported, not the other way around.
> 
> 
> You make a good point and in general I would agree with you. Isn't it
> possible, however, that there are cases where inclusion in the standard
> library would build a better community?

Not inclusion by itself, no. The standard library's APIs are only as supported
as there exists people willing to support them. *Their being in the standard
library does not create people out of thin air*. That's why the python-dev team
now have a hard requirement that new contributions must come with a guarantee of
support. Asking for inclusion without offering the corresponding guarantee will
be met with rejection, and rightly so.

> How would you or the rest of the community react to a proposal for a
> generic parallelism API? I suspect the response would be "show us an
> implementation of the code". I could whip up an implementation or adapt
> one of the existing systems, but then I worry that the discussion would
> devolve into an argument about the pros and cons of the particular
> implementation instead of the API. Even worse, it might devolve into an
> argument of the value of fine-grained vs. coarse-grained parallelism or
> the GIL. Considering that these issues seem to have been discussed quite
> a bit already and there are already multiple parallel processing
> implementations, it seems like the way forward lies in either a blessing
> of a particular package that already exists or adoption of an API
> instead of a particular implementation.

Well, you can't design a good API without having an implementation of it. If you
can't use the API in real problems, then you won't know what problems it has.
Preferably, for an API that's intended to have multiple "vendors", you should
have 2 implementations taking different approaches so you can get some idea of
whether the API generalizes well.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco