[IPython-dev] load_balanced_view

[MinRK]

On Mon, Apr 23, 2012 at 12:03, Satrajit Ghosh <satra at mit.edu> wrote:

> hi min,
>
>  ps. the reason i couldn't make it another task is because of the
>>> distributed context. the engine on which the dependent task runs cannot be
>>> controlled (i think).
>>
>>
>> Nope.  The engines can be controlled quite specifically.
>>  LoadBalancedViews have a 'targets' attribute, just like DirectViews, but
>> they select which *subset* of targets across which the tasks may be
>> distributed.  If this is a single engine, then you know exactly where it
>> will run.
>>
>
> this is more academic at this point, but just to be clear on the
> functionality.
>
> i have X engines running on X machines. and i'm distributing a task to
> some arbitrary engine using LoadBalancedView. is there a way for me to send
> task A and B and say run B on the same engine as A.
>

No,


>  i don't want to run A on any particular engine.
>

Yes, this is what the `follow` dependencies accomplish, and what Fernando
referred to in his initial comment: Controlling the *destination* of tasks,
as a function of the destination of prior tasks:

def task_one(b):
    "store b in global a, for some reason"
    global a
    a = b

def task_two(c):
    "product of a and c, must be run following task_one"
    return c*a

rc = parallel.Client()

view = rc.load_balanced_view()

# submit task_one anywhere
ar_one = view.apply_async(task_one, 5)

# submit task_two to the same engine as task_one (even though we don't know
where that is, yet)
with view.temp_flags(follow=ar_one):
    ar_two = view.apply_async(task_two, 10)

ar_two.get()

print "task one ran on engine %i" % ar_one.engine_id
print "task two ran on engine %i" % ar_two.engine_id

We call these Graph Dependencies, and you can read a bit more about them in
the docs<http://ipython.org/ipython-doc/dev/parallel/parallel_task.html#graph-dependencies>
.

Not knowing how your code actually works, it's possible this could be done
with functional dependencies (functions that run prior to the real one).
 This may not be true for you, though, as the dependency function is called
before the task is *run*, but not before it is unserialized.

> if i remember target attributes were setup during creation of an engine?

I don't know what this could refer to.  Nothing is setup on the client side
or in the schedulers during the creation of an engine.

-MinRK


>
> cheers,
>
> satra
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20120423/a65b8fd7/attachment.html>