[IPython-dev] IPython1 parallel questions

[Douglas Jones]

Hello IPython group!

I recently discovered IPython1 and its parallel capabilities. Once I
read about the feature set, I became extremely excited as IPython1
seems to solve many of the interactive parallel computing problems
that I had hoped to solve for a project that I am a developer on.

Let me start by framing my project. Our code currently exists as
library written in C++. Our library does parallel computation using
MPI. We currently have code that exposes this to python in an
interactive manner, and have been looking for ways to expand this to a
fuller more robust implementation. I'm hoping that IPython1 will be
the perfect piece to create our collaborative, interactive
environment.

That said, as I investigate IPython further and start to develop some
prototype code, I have some questions for the community and
developers.

1) Are there any outstanding issues or problems with the IPython1 code
base that inhibits parallel computation? I've been experimenting with
the code and its features for the past few days in addition to looking
through the bug list, and from what I can tell there are none. Are
there any features mentioned in IPython's documentation or the
presentations on the site that haven't been implemented yet?

2) Is there any authentication or security between the client shell
and the IPython controller?

3) What are the proper ways to configure the operation of the client
and the controller or the engines? From what I've seen it looks like
there is an API that allows the user to set certain options. Can
configuration files be used?

Some configuration issues I'm thinking of right now involve output.
For instance, is there a way to turn off the feedback from each node
for every line of code? I would imagine that if you had a cluster with
many nodes running, you would not want to see this feedback. Another
issue is how output on stdout from C++ code is handled. I noticed that
right now, any output on stdout for a simple C++ module that I have
exposed to python and loaded in each engine simply is simply written
by the engine to the console. Is there some way to redirect this
output?

4) When objects are sent from the kernel to the client, is the only
prerequisite to this that the object being sent is able to be pickled?
I suppose the client would also have to have the class code for any
objects imported?

5) Are there any known projects that use IPython1 for interactive
scientific computing? I'd be really interested to see ones that also
support visualization of distributed data.

Well, I think that covers all my initial questions. Thank you for
bearing this long post and my newbie questions. I'm really looking
forward to working more with IPython as it seems like an amazing piece
of software.

Thank you,
~doug