[IPython-dev] Before a patch for LSF support

[Brian Granger]

> I ran into another issue: on a cluster, the home folder may be
> different than on the access box. In that case, the .ipython/security
> does not exist and the engine will not start (I've just tested this).
>

Currently our model for ipcluster is that:

.ipython/security is shared by all hosts and in the same location.  If you
don't have this situation, you will have to manually move the .furl files
around and tell ipengine where the .furl files are located.  You will also
need to use persistent furl files.  Docs on all this are here:

http://ipython.scipy.org/doc/stable/html/parallel/parallel_process.html

Let us know if you have other questions - this side of things can be very
subtle.  Another thing to watch out for.  Some batch systems *require* the
processes on compute nodes to call MPI_Init upon starting.  This can be
accomplished by using mpi4py.  See how we do this in the mpiexec/mpirun
versions of ipcluster.  But on some system (depends on which MPI) that is
not enough.  Some systems require that the *VERY FIRST* things a process
does is call MPI_Init.  On these systems you will need to build a custom
version of the python binary that handles this correctly.  Again, mpi4py
provides such a binary.  Hopefully you won't have to deal with these things!

Cheers,

Brian


>
> >> I will try to setup the cluster stuff now.
> >
> > Great, looking forward to the patch!
>
> Well, first I have to try to understand what is going on, as I don't
> even get the log files after the engine submission !
>
> Also, I've tried to extract the job id (it seems it is needed), but
> the BatchEngineSet.parse_job_id extracts everything that is matched by
> the regexp describing a job (it uses group()). I had to put "Job
> <(\d+)>" as a regexp, so group() returns, for instance, "Job <1234>"
> instead of "1234". I may submit a patch to get group(1) and modify the
> PBS regexp accordingly.
>

Yes, you will *very* likely have to modify the various regexps.



>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090809/0dab3893/attachment.html>