[IPython-dev] Using IPython Cluster with SGE -- help needed

[Andreas Hilboll]

Hi,

I would like to use IPython for calculations on our cluster. It's a
total of 11 compute + 1 management nodes (all running Linux), and we're
using SGE's qsub to submit jobs. The $HOME directory is shared via NFS
between all the nodes.

Even after reading the documentation, I'm unsure about how to get things
running. I assume that I'll have to execute ``ìpcluster -n 16`` on all
compute nodes (they have 16 cores each). I'd have the ipython shell
(notebook won't work due to firewall restrictions I cannot change) on
the management node. But how does the management node know about the
kernels which are running on the compute nodes and waiting for a job?
And how can I tell the management node that it shall use qsub to submit
the jobs to the individual kernels?

As I think this is a common use case, I'd be willing to write up a nice
tutorial about the setup, but I fear I need some help from you guys to
get things running ...

Cheers,

-- Andreas.