[IPython-dev] running multiple cells simultaneously

[William Stein]

On Tue, Jul 8, 2014 at 7:42 PM, Andrew Chen <chenjandrew at gmail.com> wrote:
> I'm interested in being able to run mutiple ipython notebook cells
> simultaneously: for example, to run a loop in once cell, and use another
> loop with interactive widgets to tune some parameter of the loop in the
> first cell.
>
> I am aware of the ipython.parallel, but this seems doesn't seem perfectly
> relevant because the parallel interface is still a blocking REPL loop. The
> interaction (as I understand it) is "read -
> EvaluateOnSeparateProcesses/Kernels - PrintAllResultsFromProcesses", where I
> am interested in essentially being able to have "read-evaluate....<switch to a
> different cell> read-evaluate-print <switch back>... print".
>
> I also tried to use standard python threads, and attempted to use the advice
> from
> this thread to redirect the cell output:
> http://stackoverflow.com/questions/14393989/per-cell-output-for-threaded-ipython-notebooks
> however, it did not work as advertised, and threads have a nasty habit of
> being difficult to debug.
>
> I'm curious if this is possible, and what I would need to do/look at to
> enable this functionality.

Quick remark.  I implemented something like this a while ago for
SageMathCloud (SMC) worksheets.

You put %fork at the beginning of a cell, and that cell starts running
as a forked off subprocess.  You can continue evaluating
any other cells as normal, or making new "%fork" cells.   When the
subprocess finishes evaluation, any newly created pickle-able
variables are pickled, and set in the calling parent process.  That's it.

Using fork means that you can run multiple CPU-bound computations in parallel.

I've attached the source code for what I did for SMC, in case it is inspiring.

William



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2014-07-08 at 11.02.28 PM.png
Type: image/png
Size: 179503 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140708/4f42f693/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fork.py
Type: text/x-python-script
Size: 3065 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20140708/4f42f693/attachment.bin>