[IPython-dev] Named Engines

[Fernando Perez]

On Wed, Jul 21, 2010 at 2:35 AM, MinRK <benjaminrk at gmail.com> wrote:
> I now have my MonitoredQueue object on git, which is the three socket Queue
> device that can be the core of the lightweight ME and Task models (depending
> on whether it is XREP on both sides for ME, or XREP/XREQ for load balanced
> tasks).

Great!

> The biggest difference in terms of design between Python in the Controller
> picking the destination and this new device is that the client code actually
> needs to know the XREQ identity of each engine, and all the switching logic
> lives in the client code (if not the user exposed code) instead of the
> controller - if the client says 'do x in [1,2,3]' they actually issue 3
> sends, unlike before, when they issued 1 and the controller issued 3. This
> will increase traffic between the client and the controller, but
> dramatically reduce work done in the controller.

As best I can see, that's actually a net win, as long as we hide it
from user-visible APIs: the simpler the controller code, the less our
chances of it bottlenecking (since the controller has also other
things to do).  I really like the idea of having most of the logic
effectively embedded in the 0mq device.

> Since the engines' XREP IDs are known at the client level, and these are
> roughly any string, it brings up the question: should we have strictly
> integer ID engines, or should we allow engines to have names, like
> 'franklin1', corresponding directly to their XREP identity?
> I think people might like using names, but I imagine it could get confusing.
>  It would be unambiguous in code, since we use integer IDs and XREP
> identities must be strings, so if someone keys on a string it must be the
> XREP id, and if they key on a number it must be by engine ID.

I suspect having named IDs could be useful, as an optional feature.
People may have naming conventions for their hosts and we could expose
a way to auto-collect the hostname as default ID, and then assign
-0...-N suffixes to each engine in a multicore host (host-0, host-1,
...).

As long as internally these don't cause problems, I don't see why not have them.

Cheers,

f