Fastest first

[Cameron Simpson]

On 24Dec2018 16:29, Avi Gross <avigross at verizon.net> wrote:
>But none of this applies to my original question. I am talking about 
>whether anyone has created a general framework for a much simpler 
>scenario. Loosely speaking, I might be asked to create a list of 
>objects or functions that somehow can be started independently in the 
>same way. I hand this over to some module that takes my list and does 
>the rest and eventually returns results. That module might accept 
>additional choices and instructions that fine tune things. But the 
>overall code might look like:
>
>Import some module
>Make a new object that will run things for me as some kind of controller.
>Invoke incantations on that controller that include setting things and especially getting it the list of things to do.
>I ask it to get started and either wait or go do something else.
>At some later point I find it has completed. The results of the problem 
>are somewhere I can reach. Perhaps there are logs I can search to see 
>more details like which methods quit or were stopped or what their last 
>result was before ...

Well, I have a pair of modules named "cs.result" and "cs.later" in PyPI 
which offer this.  I imagine the futures module also does: I had 
cs.later before that arrived.

cs.result has a class called a Result, instances are objects with a 
.result attribute which may be fulfilled later; they're also callable 
which returns the .result - when it becomes ready.  cs.later is a 
capacity managed queue using these Result objects.

Basicly, you can use a cs.later.Later to submit functions to be run, or 
make cs.result.Results directly and tell them to fulfil with an 
arbitrary function. Both of these dispatch Threads to run the function.

Having submitting a bunch of functions and having the corresponding 
Results to hand, you can wait for the Results as they complete, which 
does what you're after: get the earliest result first.

Typical code looks like:

    # a Later with 4 permitted parallel tasks at a given time
    L = Later(4)
    LFs = []
    # get a LateFunction from the Later (a Result subclass)
    LF = Later.defer(function, *args, **kwargs)
    LFs.append(LF)

You can submit as many as you like, and keep them in a list for example 
such as "LFs" above.

There's a report generator function which takes a collection of Results 
and yields them as they complete.

    for LF in report(LFs):
        print(LF, "completed")

You can clearly build on this depending on your use case.

The typical implementation uses Threads, but of course a Thread can run 
a subprocess if that's sensible, and so forth, or you can use some other 
system to fulfil the Results; from the outside the user doesn't care.

Cheers,
Cameron Simpson <cs at cskk.id.au>