[IPython-dev] pushAll and executeAll speed issue

[Albert Strasheim]

Hello all

I'm doing some experiments with IPython1 on Windows. I'm using DeinoMPI's 
GUI on top of mpi4py to start two kernels on my Core 2 Duo system. They 
connect to the controller without problems. I make a RemoteController and 
off I go. This is too cool.

(By the way: as I understand it, MPI is currently only being used to start 
the kernels. From what I can see, it is not yet being used inside things 
like pushAll, etc. Is this the next step to use the MPI functions if they 
are available? Anybody working on this? If not, maybe I can try my hand at 
it. Any hints or suggestions on where to start appreciated.)

Anyway, back to my experiment. The script I'm using is attached.

It starts off with some local benchmarks running on a single core. I get:

cPickle dumps time 4.95649055955
cPickle loads time 2.39126201794
local map time 10.3841574837

Then I push to all and time it. Here I get:

total pushAll time 70.6287321433

I was expecting something proportional to dumpsTime+N*loadsTime (if I 
understand correctly that it is serially pushing out the data to each of the 
N kernels) or maybe even less. I noticed that this section gets dramatically 
slower even with a relatively small increase in data size. While this is 
busy, I see Python using up all the CPU on one core as expected, but I also 
see that the Windows "System" process is very busy.

Next up, the timing over the executeAll call that does the map:

executeAll time for map 139.186242386

but the kernels print:

engine map time 12.8248084603
engine map time 11.3900467797

which is only slightly more than the single core time. However, I have to 
wonder where the heck the rest of the time went?! :-)

Do you guys see similar timings on Linux? Any thoughts on where my time is 
going? Could it be Twisted? I tried with the latest from SVN and the 
official 2.4.0 release for Windows.

Cheers,

Albert 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: frags9.py
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061109/d17ae91c/attachment.ksh>