[SciPy-user] Parallel OO programming in Python.
Flavio Coelho
fccoelho at gmail.com
Mon Oct 30 14:47:10 EST 2006
Thanks Brian,
I am playing with it right now! I am planning to run it soon on multiple
machines with multiple CPUs. Will I have to install Ipython1 on every node
or Will the controller host spawn the processes across the net? In the
README you seem to be starting the engines locally on each host....
but before I get Ahead of myself , I am getting the following error when I
try to start an engine on my localhost:
fccoelho at sombra ~/Downloads/Twisted $ ipengine &
[2] 32018
fccoelho at sombra ~/Downloads/Twisted $ 2006/10/30 16:44 -0200 [-] Log opened.
2006/10/30 16:44 -0200 [-] Starting factory <
ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineServiceobject
at 0xb6706eac>
2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] Unhandled
Error
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/twisted/python/log.py",
line 48, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.4/site-packages/twisted/python/log.py",
line 33, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.4/site-packages/twisted/python/context.py",
line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args,
**kw)
File "/usr/lib/python2.4/site-packages/twisted/python/context.py",
line 37, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File
"/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line
139, in _doReadOrWrite
why = getattr(selectable, method)()
File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py",
line 362, in doRead
return self.protocol.dataReceived(data)
File
"/usr/lib/python2.4/site-packages/twisted/protocols/basic.py", line 99, in
dataReceived
self.doData()
File
"/usr/lib/python2.4/site-packages/twisted/protocols/basic.py", line 62, in
doData
self.stringReceived(self.__buffer)
File
"/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line
557, in stringReceived
self.nextHandler(msg)
File
"/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line
573, in dispatch
f(args)
File
"/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line
706, in handle_REGISTER
qe = engineservice.QueuedEngine(self, keepUpToDate=True)
File
"/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line
342, in __init__
self.registerMethods()
File
"/usr/lib/python2.4/site-packages/ipython1/kernel/engineservice.py", line
345, in registerMethods
zi.alsoProvides(self, *self.engine.__provides__)
exceptions.AttributeError: 'module' object has no attribute
'alsoProvides'
2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1]
unregistered engine None
2006/10/30 16:44 -0200 [VanillaEngineClientProtocol,client] Stopping factory
<ipython1.kernel.enginevanilla.VanillaEngineClientFactoryFromEngineServiceobject
at 0xb6706eac>
2006/10/30 16:44 -0200 [VanillaEngineServerProtocol,0,127.0.0.1] Unhandled
Error
Traceback (most recent call last):
File "/usr/bin/ipcontroller", line 51, in main
reactor.run()
File
"/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 218,
in run
self.mainLoop()
File
"/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 229,
in mainLoop
self.doIteration(t)
File
"/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line
133, in doSelect
_logrun(selectable, _drdw, selectable, method, dict)
--- <exception caught here> ---
File "/usr/lib/python2.4/site-packages/twisted/python/log.py",
line 48, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.4/site-packages/twisted/python/log.py",
line 33, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.4/site-packages/twisted/python/context.py",
line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args,
**kw)
File "/usr/lib/python2.4/site-packages/twisted/python/context.py",
line 37, in callWithContext
return func(*args,**kw)
File
"/usr/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line
149, in _doReadOrWrite
self._disconnectSelectable(selectable, why, method=="doRead")
File
"/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 256,
in _disconnectSelectable
selectable.connectionLost(failure.Failure(why))
File "/usr/lib/python2.4/site-packages/twisted/internet/tcp.py",
line 416, in connectionLost
protocol.connectionLost(reason)
File
"/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line
538, in connectionLost
self.factory.unregisterEngine(self.id)
File
"/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py", line
1023, in unregisterEngine
return self.service.unregisterEngine(id)
File
"/usr/lib/python2.4/site-packages/ipython1/kernel/controllerservice.py",
line 300, in unregisterEngine
del self.engines[id]
exceptions.KeyError:
On 10/30/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
>
> > Can Ipython1 take advantage of multiple core cpus or SMP machines in the
> > same way it does for a networked cluster? It would be nice to have that
> > since quad-core cpu are arriving on the market and SMP machines are
> pretty
> > common already.
>
> This was one of the major usage cases we have had in mind. So yes.
> Some details though:
>
> In IPython1 computations are done by a separate process that we are
> calling the IPython engine. Multiple IPython engines can be run on
> multiple systems (distributed memory), a single (multi CPU and/or
> multi-core) system or a combination of the two. I have run two
> IPython engines on my dual core MacBook and seen 2x speedup with
> simple python scripts. I wasn't using MPI, but with MPi you should
> see a similar behavior as long as your algorithm scales linearly to 2
> instances processes.
>
> I can't wait to use IPython1 with 4/8/16 cores on my workstation :)
>
> Please let us know if you have questions or problems.
>
> Brian
>
> On 10/30/06, Flavio Coelho <fccoelho at gmail.com> wrote:
> > Hi Brian,
> >
> > Can Ipython1 take advantage of multiple core cpus or SMP machines in the
> > same way it does for a networked cluster? It would be nice to have that
> > since quad-core cpu are arriving on the market and SMP machines are
> pretty
> > common already.
> >
> > Thanks!
> >
> > Flávio
> >
> >
> > On 10/30/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > > I also looked at the entry on Ipython1 and it looks really
> promising.
> > > > Did I understand correctly that Ipython1 is able to run parallel
> python
> > > > code, regardless of which "MPI for python" package one eventually
> uses?
> > >
> > > In principle yes. The IPython engines have a configuration option
> > > that you use to import the mpi bindings of your choice. But, to work
> > > properly, the import statement must cause MPI_Init() to be called. If
> > > the bindings you are using don't do that, it should b easy to write a
> > > wrapper module that imports and also calls MPI_Init(). Also, some of
> > > this depends on what MPI implementation you are using. The more
> > > moderns ones (like openmpi) are really nice and pretty forgiving. The
> > > older ones, though, sometimes require that Python itself (not your
> > > script) call MPI_Init(). This is also very easy to work around. The
> > > bottom line is that we can help you get it to work with whatever mpi
> > > binding and implementation you want.
> > >
> > > Withh that said, I highly do highly recommend mpi4py and openmpi - we
> > > use these regularly with ipython and it all works well.
> > >
> > > Brian
> > >
> > >
> > > > Giovanni
> > > >
> > > > Flavio Coelho wrote:
> > > > > Thank you for this post and many thanks to Fernando Perez, for the
> > > > > awesome tool that Ipython1 is becomming!!! I am beginning to use
> it
> > > > > today!!
> > > > >
> > > > > Flávio
> > > > >
> > > > > On 10/29/06, * Brian Granger* < ellisonbg.net at gmail.com
> > > > > <mailto:ellisonbg.net at gmail.com>> wrote:
> > > > >
> > > > > > For object oriented parallel programming in Python on a
> single
> > > > > machine I can use techniques such as Bulk Synchronous
> Parallelism
> > > > > (BSP) or Message Passing Interface (MPI). There are mentioned
> some
> > > > > modules and packages for Python on
> > > > > > http://www.scipy.org/Topical_Software
> > > > > > Article: Parallel and distributed programming.
> > > > > > After reading prerequisites and limitations I thing that the
> > > > > following ones could be good for me:
> > > > > > PyMPI
> > > > > > Pypar
> > > > > > MPI for Python
> > > > >
> > > > > I think mpi4py is the best option. It is a very nice
> > implementation
> > > > > and the developer is working really hard to make it a great
> > package.
> > > > > Also, I don't think pympi and pypar are being developed
> actively
> > > > > anymore.
> > > > >
> > > > > Another thing you might check out is IPython's parallel
> computing
> > > > > facilities that let you use mpi4py (and other mpi bindings)
> > > > > interactively:
> > > > >
> > > > > http://ipython.scipy.org/moin/Parallel_Computing
> > > > >
> > > > > I have never used BPS, but am familiar with the model. Can
> > probably
> > > > > get it working interactively with IPython as well.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Brian
> > > > >
> > > > > > But I can't distinguish which one brings fewer limitations,
> is
> > > > > efficient in application and will develop in future.
> > > > > > My favorite is BSP, but I can't find a package for present
> SciPy
> > > > > and NumPy.
> > > > > >
> > > > > > Could anybody give me a recommendation, which way I should
> go.
> > > > > >
> > > > > > Thanks.
> > > > > > Fie Pye
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ---------------------------------------
> > > > > > Máme další spot P.Čtvrtníčka. Podívejte se www.neuservis.cz
> > > > > <http://www.neuservis.cz>
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > SciPy-user mailing list
> > > > > > SciPy-user at scipy.org <mailto:SciPy-user at scipy.org>
> > > > > >
> > http://projects.scipy.org/mailman/listinfo/scipy-user
> > > > >
> > <http://projects.scipy.org/mailman/listinfo/scipy-user >
> > > > > >
> > > > > _______________________________________________
> > > > > SciPy-user mailing list
> > > > > SciPy-user at scipy.org <mailto: SciPy-user at scipy.org>
> > > > >
> > http://projects.scipy.org/mailman/listinfo/scipy-user
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Flávio Codeço Coelho
> > > > > registered Linux user # 386432
> > > > > ---------------------------
> > > > > "Laws are like sausages. It's better not to see them being made."
> > > > > Otto von Bismark
> > > > >
> > ------------------------------------------------------------------------
> > > > >
> > > > > _______________________________________________
> > > > > SciPy-user mailing list
> > > > > SciPy-user at scipy.org
> > > > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > > > >
> > > > _______________________________________________
> > > > SciPy-user mailing list
> > > > SciPy-user at scipy.org
> > > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > > >
> > > _______________________________________________
> > > SciPy-user mailing list
> > > SciPy-user at scipy.org
> > > http://projects.scipy.org/mailman/listinfo/scipy-user
> > >
> >
> >
> >
> > --
> >
> > Flávio Codeço Coelho
> > registered Linux user # 386432
> > ---------------------------
> > "Laws are like sausages. It's better not to see them being made."
> > Otto von Bismark
> > _______________________________________________
> > SciPy-user mailing list
> > SciPy-user at scipy.org
> > http://projects.scipy.org/mailman/listinfo/scipy-user
> >
> >
> >
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>
--
Flávio Codeço Coelho
registered Linux user # 386432
---------------------------
"Laws are like sausages. It's better not to see them being made."
Otto von Bismark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20061030/87b2eeb9/attachment.html>
More information about the SciPy-User
mailing list