Addition of multiprocessing ill-advised? (was: Python 3.0.1)

[Jesse Noller]

On Tue, Jan 27, 2009 at 10:49 PM, Ben Finney <ben at benfinney.id.au> wrote:
> (Continuing a side topic of a different discussion)
>
> Steve Holden <steve at holdenweb.com> writes:
>
>> I think that [Python 2.6 was a rushed release]. 2.6 showed it in the
>> inclusion (later recognizable as somewhat ill-advised so late in the
>> day) of multiprocessing […]
>
> What was ill-advised about the addition of the 'multiprocessing'
> module to Python 2.6? I ask because I haven't yet used it in anger,
> and am not sure what problems have been found in it.
>
> --
>  \                  "Holy bouncing boiler-plated fits, Batman!" —Robin |
>  `\                                                                   |
> _o__)                                                                  |
> Ben Finney
> --
> http://mail.python.org/mailman/listinfo/python-list
>

I might write a longer blog post about this later, but I can see
Steve's point of view. The fact is, pyprocessing/multiprocessing was a
late addition to Python 2.6. Personally, I was game to put it into
either 2.7 or 2.6, but I felt inclusion into 2.6 wasn't completely out
of question - and others agreed with me.

See these mail threads:

http://mail.python.org/pipermail/python-dev/2008-May/079417.html
http://mail.python.org/pipermail/python-dev/2008-June/080011.html

And so on.

All of that being said; the initial conversion and merging of the code
into core exposed a lot of bugs I and others didn't realize were there
in the first place. I take full responsibility for that - however some
of those bugs were in python-core itself (deadlock after fork
anyone?).

So, the road to inclusion was a bit rougher than I initially thought -
I relied heavily on the skills of people who had more experience in
the core than I did, and it was disruptive to the release schedule of
python 2.6 due to both the bugs and instability.

I however; disagree that this was ultimately a bad decision, or that
it was some how indicative of a poorly managed or rushed 2.6 release.
All releases have bugs, and towards the end of the 2.6 cycle,
multiprocessing *was not* the release blocker.

After 2.6 went out, I had a small wave of bugs filed against
multiprocessing that I've been working through bit by bit (I still
need to work on BSD/Solaris issues) and some of the bugs have exposed
issues I simply wish weren't there but I think this is true of any
package, especially one as complex as multiprocessing is.

I know of plenty of people using the package now, and I know of
several groups switching to 2.6 as quickly as possible due to its new
features, bug fixes/etc. Multiprocessing as a package is not bug free
- I'm the first to admit that - however it is useful, and being used
and frankly, I maintain that it is just one step in a larger project
to bring additional concurrency and distributed "stuff" into
python-core over time.

So yes, I see Steve's point - multiprocessing *was* disruptive, and it
inclusion late in the game siphoned off resources that could have been
used elsewhere. Again, I'll take the responsibility for soiling the
pool this way. I do however think, that python 2.6 is overall a
*fantastic* release both feature wise, quality wise and is quite
useful for people who want to "get things done" (tm).

Now I'm going to go back to fixing bugs.

-jesse