[IPython-dev] Kernels and parallel support?

[Doug Blank]

On Sat, Mar 22, 2014 at 1:59 PM, MinRK <benjaminrk at gmail.com> wrote:
> On Sat, Mar 22, 2014 at 6:47 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>
>> On Fri, Mar 21, 2014 at 12:10 PM, Min RK <benjaminrk at gmail.com> wrote:
>> >
>> >> On Mar 21, 2014, at 5:36, Doug Blank <doug.blank at gmail.com> wrote:
>> >>
>> >>> On Thu, Mar 20, 2014 at 4:06 PM, Doug Blank <doug.blank at gmail.com>
>> >>> wrote:
>> >>>> On Thu, Mar 20, 2014 at 2:19 PM, MinRK <benjaminrk at gmail.com> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>>> On Thu, Mar 20, 2014 at 10:58 AM, Doug Blank <doug.blank at gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> After reading:
>> >>>>>
>> >>>>> http://ipython.org/ipython-doc/dev/parallel/parallel_intro.html
>> >>>>>
>> >>>>> and related pages, I am unsure what the relationship between
>> >>>>> parallel
>> >>>>> processing and kernels are. Can external kernels be used in a
>> >>>>> parallel
>> >>>>> environment? Or is this part of IPython python-kernel specific? Does
>> >>>>> it use ZMQ messages?
>> >>>>
>> >>>>
>> >>>> Depends what you mean by 'external'. You can start engines at any
>> >>>> time on
>> >>>> any machine, and they can join a cluster.  Yes, IPython.parallel uses
>> >>>> zmq.
>> >>>
>> >>> If I wanted to explore using a cluster of 3rd-party kernels, such as
>> >>> IHaskell or ICalico, in a parallel environment, where would I begin?
>> >>> Has anyone done this before?
>> >>>
>> >>> I see here:
>> >>>
>> >>>
>> >>> http://ipython.org/ipython-doc/dev/parallel/magics.html#engines-as-kernels
>> >>>
>> >>> that kernels connect to a controller to create an engine. If the
>> >>> standard ipcontrollers can start up external kernels, I guess a place
>> >>> to begin might be implementing what the %px magic does?
>> >>
>> >> Along with those general questions, a specific one:
>> >>
>> >> I would have thought that "ipcluster start --profile calico" would
>> >> have used my c.KernelManager.kernel_cmd to start my kernels, but I
>> >> can't get ipcluster to use it. Likewise, if I use the "Cluster" tab in
>> >> the notebook to start a cluster, I also don't get my kernel.
>> >>
>> >> How to start a cluster of third-party kernels?
>> >
>> > It's not that simple. KernelManagers are not used in ipcluster.
>> > Normally, all engines are started manually (ipcluster is basically: for i in
>> > range(n): ipengine).  If you want to turn your kernel into an engine, you
>> > are going to have to teach it about the ipcontroller connection files, and
>> > connecting instead of binding.
>>
>> First, the idea of allowing non-Python kernels to be able to easily
>> become parallelized is a fantastic possibility! There are many
>> languages/systems that could benefit from this.
>>
>> I have gone through the anatomy of the communication between a hub,
>> engines, and a notebook here:
>>
>> http://wiki.roboteducation.org/IPython_Parallel_API
>>
>> (corrections of misunderstandings welcome). Two questions:
>>
>> 1) In addition to adding the ipcontroller communications needed, it
>> appears that pickling and eval are used to execute remote code (even
>> though pack/unpack settings in ipcontroller_*.py files are marked as
>> "json"). Of course, this won't work for non-Python kernels. Could a
>> real json representation be used for values, getting rid of the need
>> for pickle?
>
>
> JSON *is* used for message serialization. However, the main message used in
> IPython.parallel is `apply`, which sends native functions and data, and thus
> uses native serialization (pickle on Python). There are no plans to change
> this. Code-as-text execution (%px, view.execute), however, uses the same
> execute_request message that is used for execution in the notebook, etc.,
> and should work fine on a non-Python kernel.

Ok, no need to change anything as long as there is a method to
evaluate code-as-text. Sounds perfect!

>>
>>
>> 2) Knowing that "Full, integrated support for non-Python kernels" is a
>> line item in the 3.0 roadmap [1], and this is mentioned in
>> IPython/parallel/engine/engine.py:
>>
>> # FIXME: This is a hack until IPKernelApp and IPEngineApp can be fully
>> merged
>> app = IPKernelApp(parent=self, shell=self.kernel.shell,
>> kernel=self.kernel, log=self.log)
>>
>> is there a chance that Kernels and Engines will be merged this year
>> so that parallel processing for non-Python kernels would be possible?
>
>
> Yes, this is my hope.

Excellent! As you know, I've started issue #5409 to track this
request. Thanks again!

-Doug

https://github.com/ipython/ipython/issues/5409

>>
>>
>> Thanks for any additional information!
>>
>> -Doug
>>
>> [1] -
>> https://hackpad.com/IPython-Winter-2014-Development-Meeting-fKrExqKCWmC
>>
>> >
>> >>
>> >> -Doug
>> >>
>> >>> -Doug
>> >>>
>> >>>> -MinRK
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>> Thanks for any additional data,
>> >>>>>
>> >>>>> -Doug
>> >>>>> _______________________________________________
>> >>>>> IPython-dev mailing list
>> >>>>> IPython-dev at scipy.org
>> >>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> IPython-dev mailing list
>> >>>> IPython-dev at scipy.org
>> >>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >>>>
>> >> _______________________________________________
>> >> IPython-dev mailing list
>> >> IPython-dev at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>