[SciPy-user] Multiprocessing, GUIs and IPython

[Robin]

On Wed, Jan 7, 2009 at 8:00 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Hi,
>
> I see that people are starting to use multiprocessing to parallelize
> numerical Python code.  I am wondering if we want to allow/recommend
> using multiprocessing in scipy.  Here are some of my concerns:
>
> * Currently multiprocessing doesn't play well with IPython.  Thus, if
> scipy starts to use multiprocessing, people will get very unpleasant
> surprises when using IPython.  I don't know exactly what the problems
> are, but my feeling is that it is unlikely that IPython will ever have
> *full* support for multiprocessing.  Some support might be possible,
> though.

I've used multiprocessing (or actually pyprocessing) a little bit with
IPython. The main problem is that you can't use interactively defined
functions - ie can't do something like
p = Pool(8)
p.map(lambda x: somefunc(x,2,3), range(1,100))

because pickling interactively defined stuff doesn't work in Ipython.
Other than that though - it's been working fine for me, (ie just make
sure anything you are using is defined in a module so pickle works):

from module import somefunc
p.map(somefunc, range(1,100)

although I haven't been trying to do anything too clever (haven't had
any plots open or anything like that)

I think it adds a very valuable feature - that for a beginner like me
is much easier to get to grips with than MPI or even the clustering
features of ipython, to easily allow use of multi-core machines.

It would be great if IPython could sort out the pickle business so you
could pickle interactively defined functions (they currently don't
show up in __main__ which is a FakeModule instance).

Robin