[SciPy-user] Multiprocessing, GUIs and IPython

[Brian Granger]

> Too late! I use it in almost all code :).

I don't care if you use multiprocessing in your own code - I am
thinking only about numpy/scipy here.

> OK, none if this is in Scipy,
> but multiprocessing is starting to creep in various places.

> As Robert points out, that's because of wizardry done by IPython. That's
> really a pity, because in my experience, multiprocessing is fairly
> robust. Nothing that's not fixable from IPython's side, though, I
> believe.

Yes, a bug report should be added to IPython's launchpad site about this.

>> * Multiprocessing doesn't play well with other things as well, such as
>> Twisted.  Again, if scipy uses multiprocessing, it would become
>> usuable within Twisted based servers.
>
> IMHO that's a bug of Twisted :).

Then please file a bug report with Twisted :)

More seriously, Twisted has been around *a bit* longer than
multiprocessing and is much better tested in both the unittest sense
and in the real world sense.  The informal word from the Twisted
community is that there are fundamental incompatabilities between
Twisted and multiprocessing and that in no way are these
incompatabilities in the "Twisted bug category."  But, I do hope these
things are eventually worked out.

> More seriously, multiprocessing is now
> in the standard library. It may have some quirks, but I think everybody
> should try and play well with it, and I wouldn't be surprised to see
> things improving as people get familiar with it.

Yes, because it is in the standard library, we should all try to play
well with it.  And I do hope things improve.  However,
multiprocessing's implementation (as I understand it) carries some
strong constraints that exclude certain potential friends (like
Twisted).

> I must admit I would really like to see IPython using multiprocessing as
> a backend for single-computer parallel computing (I have 8 cores, so I do
> a lot of that). I don't know if it is compatible with IPython's
> architecture. Specifically, I would like to be able to use the same API
> than IPython, with a fork-based mechanism. I would also like the easy
> process management.

Because of multiprocessing's inability to play well with Twisted, this
exact thing probably won't happen - at least anytime soon.  However,
it is very possible that IPython might have a multiprocessing-like
API.