[IPython-dev] Parallel: Report progress from the engines

[Darlan Cavalcante Moreira]

Hi list,

I'm using the fantastic parallel computing machinery from IPython, which
works really well. However, the simulations I'm performing take a few
hours to finish and I'd like to see the progress from each engine. The
number of tasks is small (usually from five to eight) such that only
knowing how many tasks have finished is not very useful for me. I need
to track the progress of each individual task.

To give an example, with the multiprocessing module I could create a
Queue that I can pass as argument to the task that will run in the other
process. My tasks are basically a for loop running a few lines of code
in each iteration. Therefore, the task can put the iteration number in
the Queue indicating its progress, which is then read (every 5 seconds)
from another process that finally prints the progress in the terminal.

However, I have no idea how I can do this with the IPython parallel
machinery. Is there any way for the engines to send data back (to the
controller?) that I can use to track their progress?


-- 
Darlan Cavalcante