[IPython-dev] Parallel map

[Gael Varoquaux]

On Sat, Mar 08, 2008 at 02:28:02AM -0800, Fernando Perez wrote:
> We need to summarize the recent work done at the sage/scipy/ipython
> sprint.

I did see some recent commits on Trac :->.

> In particular, Min did a lot of excellent work *precisely* on
> this issue, most (if not all) of which is already committed, to
> provide a full ipython daemon for process control.  This allows you to
> do exactly that, to create/control/destroy engines and/or controllers
> from within python scripts.

Great. Now a very important question: can the daemon be started from
Python without an os.system apparent for the user? I want to be able to
log in a box, and run a pPthon script that does parallel computing with
only a few lines of Python code.

> The good thing is that you're already a bzr launchpad team member, so
> I'm sure you'll soon be contributing this code and docs :)

Hum. Currently I am physicaly killing myself doing way to many things.
No, seriously, its worst than it has ever been, and I have a dangerously
off balance sleep pattern. Maybe I'll get better next year when I start
my new job. Maybe I'll even use ipython1 as a full part of my day work.

> We finally figured out a trick using the 'with' statement to allow you
> to write code like

> with all_engines:
>   do_in_remote_engines(..)

That's really nice. Now a big question is, does that remove the need for
all the objects in the parallel code to be picklable. Because that is
what was limiting me here.

Cheers,

Gaël