[Baypiggies] clustering

[Bryan O'Sullivan]

On Wed, 2006-08-30 at 14:42 -0700, Shannon -jj Behrens wrote:

> * Do I have to run a particular Linux distro?  Do they all have to be
> the same, or can I just setup a daemon on each machine?

It's a big help if all of your cluster machines are similar, unless you
want to go insane debugging differences in output due to seemingly
insignificant local configuration differences.

I'd recommend taking a look at the Rocks clustering distribution.  It's
based on CentOS, with some features to make PC clustering and
administration less painful.

> * What does "Beowulf" do for me?

Nothing.  It used to be a clustering technology for Linux boxes, back in
the days of Pentium Pro class hardware, but hasn't existed in any
meaningful form in years.  It lives on as a zombie term without any real
semantics :-)

> * How do I admin all the boxes without having to enter the same command n times?

Google for "parallel ssh".  A decent clustering distro (Rocks or Oscar)
will have one bundled.

> * I've heard that MPI is good and standard.  Should I use it?

Depends on what you want to do.  It's just a standard for message
passing library and runtime, the API providing some features for
managing group communications.  It's only a bit more abstract than raw
sockets, so you should expect plenty of deadlocks, wrong answers, and
performance problems as you develop your MPI programs :-)

>   Can I
> use it with Python programs?

Google for "Python MPI" :-)

> * Is there anything better than NFS that I could use to access the data?

If you don't mind living on the utterly bleedingest of edges, there are
dedicated clustering filesystems like GFS and Lustre available.  There's
also a standard MPI interface for I/O, called MPI-IO.  If you have a
small cluster (16 nodes is small) and not much data (16GB isn't much),
you should consider just preloading the data onto local disk on each
node and forget about network filesystems altogether.  The cluster
filesystems require you to perform intimate and unnatural acts that I
suspect you may not enjoy.

	<b