[SciPy-user] Parallel OO programming in Python.

[Brian Granger]

> Can Ipython1 take advantage of multiple core cpus or SMP machines in the
> same way it does for a networked cluster? It would be nice to have that
> since quad-core cpu are arriving on the market and SMP machines are pretty
> common already.

This was one of the major usage cases we have had in mind.  So yes.
Some details though:

In IPython1 computations are done by a separate process that we are
calling the IPython engine.  Multiple IPython engines can be run on
multiple systems (distributed memory), a single (multi CPU and/or
multi-core) system or a combination of the two.  I have run two
IPython engines on my dual core MacBook and seen 2x speedup with
simple python scripts.  I wasn't using MPI, but with MPi you should
see a similar behavior as long as your algorithm scales linearly to 2
instances processes.

I can't wait to use IPython1 with 4/8/16 cores on my workstation :)

Please let us know if you have questions or problems.

Brian

On 10/30/06, Flavio Coelho <fccoelho at gmail.com> wrote:
> Hi Brian,
>
> Can Ipython1 take advantage of multiple core cpus or SMP machines in the
> same way it does for a networked cluster? It would be nice to have that
> since quad-core cpu are arriving on the market and SMP machines are pretty
> common already.
>
> Thanks!
>
> Flávio
>
>
> On 10/30/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > I also looked at the entry on Ipython1 and it looks really promising.
> > > Did I understand correctly that Ipython1 is able to run parallel python
> > > code, regardless of which "MPI for python" package one eventually uses?
> >
> > In principle yes.  The IPython engines have a configuration option
> > that you use to import the mpi bindings of your choice.  But, to work
> > properly, the import statement must cause MPI_Init() to be called.  If
> > the bindings you are using don't do that, it should b easy to write a
> > wrapper module that imports and also calls MPI_Init().  Also, some of
> > this depends on what MPI implementation you are using.  The more
> > moderns ones (like openmpi) are really nice and pretty forgiving.  The
> > older ones, though, sometimes require that Python itself (not your
> > script) call MPI_Init().  This is also very easy to work around.  The
> > bottom line is that we can help you get it to work with whatever mpi
> > binding and implementation you want.
> >
> > Withh that said, I highly do highly recommend mpi4py and openmpi - we
> > use these regularly with ipython and it all works well.
> >
> > Brian
> >
> >
> > > Giovanni
> > >
> > > Flavio Coelho wrote:
> > > > Thank you for this post and many thanks to Fernando Perez, for the
> > > > awesome tool that Ipython1 is becomming!!! I am beginning to use it
> > > > today!!
> > > >
> > > > Flávio
> > > >
> > > > On 10/29/06, * Brian Granger* < ellisonbg.net at gmail.com
> > > > <mailto:ellisonbg.net at gmail.com>> wrote:
> > > >
> > > >     > For object oriented parallel programming in Python on a single
> > > >     machine I can use techniques such as Bulk Synchronous Parallelism
> > > >     (BSP) or Message Passing Interface (MPI). There are mentioned some
> > > >     modules and packages for Python on
> > > >     > http://www.scipy.org/Topical_Software
> > > >     > Article: Parallel and distributed programming.
> > > >     > After reading prerequisites and limitations I thing that the
> > > >     following ones could be good for me:
> > > >     > PyMPI
> > > >     > Pypar
> > > >     > MPI for Python
> > > >
> > > >     I think mpi4py is the best option.  It is a very nice
> implementation
> > > >     and the developer is working really hard to make it a great
> package.
> > > >     Also, I don't think pympi and pypar are being developed actively
> > > >     anymore.
> > > >
> > > >     Another thing you might check out is IPython's parallel computing
> > > >     facilities that let you use mpi4py (and other mpi bindings)
> > > >     interactively:
> > > >
> > > >     http://ipython.scipy.org/moin/Parallel_Computing
> > > >
> > > >     I have never used BPS, but am familiar with the model.  Can
> probably
> > > >     get it working interactively with IPython as well.
> > > >
> > > >     Cheers,
> > > >
> > > >     Brian
> > > >
> > > >     > But I can't distinguish which one brings fewer limitations, is
> > > >     efficient in application and will develop in future.
> > > >     > My favorite is BSP, but I can't find a package for present SciPy
> > > >     and NumPy.
> > > >     >
> > > >     > Could anybody give me a recommendation, which way I should go.
> > > >     >
> > > >     > Thanks.
> > > >     > Fie Pye
> > > >     >
> > > >     >
> > > >     >
> > > >     >
> > > >     > ---------------------------------------
> > > >     > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz
> > > >     <http://www.neuservis.cz>
> > > >     >
> > > >     >
> > > >     > _______________________________________________
> > > >     > SciPy-user mailing list
> > > >     > SciPy-user at scipy.org <mailto:SciPy-user at scipy.org>
> > > >     >
> http://projects.scipy.org/mailman/listinfo/scipy-user
> > > >
> <http://projects.scipy.org/mailman/listinfo/scipy-user >
> > > >     >
> > > >     _______________________________________________
> > > >     SciPy-user mailing list
> > > >     SciPy-user at scipy.org <mailto: SciPy-user at scipy.org>
> > > >
> http://projects.scipy.org/mailman/listinfo/scipy-user
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Flávio Codeço Coelho
> > > > registered Linux user # 386432
> > > > ---------------------------
> > > > "Laws are like sausages. It's better not to see them being made."
> > > > Otto von Bismark
> > > >
> ------------------------------------------------------------------------
> > > >
> > > > _______________________________________________
> > > > SciPy-user mailing list
> > > > SciPy-user at scipy.org
> > > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > > >
> > > _______________________________________________
> > > SciPy-user mailing list
> > > SciPy-user at scipy.org
> > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > >
> > _______________________________________________
> > SciPy-user mailing list
> > SciPy-user at scipy.org
> > http://projects.scipy.org/mailman/listinfo/scipy-user
> >
>
>
>
> --
>
> Flávio Codeço Coelho
> registered Linux user # 386432
> ---------------------------
> "Laws are like sausages. It's better not to see them being made."
> Otto von Bismark
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>
>
>