[stdlib-sig] futures - a new package for asynchronous execution

[Jeffrey Yasskin]

I am very happy with those changes.

I think deadlock should be addressed before the first release as it
changes the detailed semantics of some of the operations, but you've
promised to do that, so cool. :)

I think it's fine to leave the embedding of Deferred-like things into
futures and the embedding of futures into Deferred-like things until a
later release. I expect it to be requested, but I don't think it'll be
hard to add later.

On Thu, Nov 12, 2009 at 6:38 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
>
> I compiled a summary of people's feedback (about technical issues - I agree
> that the docs could be better but agreeing on the API seems like the first
> step) and have some API change proposals.
>
> Here is a summary of the feedback:
> - Use Twisted Deferreds rather than Futures
> - The API too complex
> - Make Future a callable and drop the .result()/.exception() methods
> - Remove .wait() from Executor
> - Make it easy to process results in the order of completion rather than in
> the order that the futures were generated
> - Executor context managers should wait until their workers complete before
> exiting
> - Extract Executor.map, etc. into separate functions/modules
> - FutureList has too many methods or is not necessary
> - Executor should have an easy way to produce a single future
> - Should be able to wait on an arbitrary list of futures
> - Should have a way of avoiding deadlock (will follow-up on this separately)
>
> Here is what I suggest as far as API changes (the docs suck, I'll polish
> them when we reach consensus):
>
> FutureList is eliminated completely.
>
> Future remains unchanged - I disagree that Deferreds would be better, that
> .exception() is not useful, and that .result() should be renamed .get() or
> .__call__(). But I am easily persuadable :-)
>
> The Executor ABC is simplified to only contain a single method:
>
> def Executor.submit(self, fn, *args, **kwargs) :
>
> Submits a call for execution and returns a Future representing the pending
> results of fn(*args, **kwargs)
>
> map becomes a utility function:
>
> def map(executor, *iterables, timeout=None)
>
> Equivalent to map(func, *iterables) but executed asynchronously and possibly
> out-of-order. The returned iterator raises a TimeoutError if __next__() is
> called and the result isn’t available after timeout seconds from the
> original call to run_to_results(). If timeout is not specified or None then
> there is no limit to the wait time. If a call raises an exception then that
> exception will be raised when its value is retrieved from the iterator.
>
> wait becomes a utility function that can wait on any iterable of Futures:
>
> def wait(futures, return_when=ALL_COMPLETED)
>
> Wait until the given condition is met for the given futures. This method
> should always be called using keyword arguments, which are:
>
> timeout can be used to control the maximum number of seconds to wait before
> returning. If timeout is not specified or None then there is no limit to the
> wait time.
>
> return_when indicates when the method should return. It must be one of the
> following constants:
>
>    NEXT_COMPLETED
>    NEXT_EXCEPTION
>    ALL_COMPLETED
>
> a new utility function is added that iterates over the given Futures and
> returns the as they are completed:
>
> def itercompleted(futures, timeout=None):
>
> Returns an iterator that returns a completed Future from the given list when
> __next__() is called. If no Futures are completed then __next__() is called
> then __next__() waits until one does complete. Raises a TimeoutError if
> __next__() is called and no completed future is available after timeout
> seconds from the original call.
>
> The URL loading example becomes:
>
> import functools
> import urllib.request
> import futures
>
> URLS = ['http://www.foxnews.com/',
>        'http://www.cnn.com/',
>        'http://europe.wsj.com/',
>        'http://www.bbc.co.uk/',
>        'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
>    return urllib.request.urlopen(url, timeout=timeout).read()
>
> with futures.ThreadPoolExecutor(50) as executor:
>  fs = [executor.submit(load_url, url, timeout=30) for url in URLS]
>
> for future in futures.itercompleted(fs):
>    if future.exception() is not None:
>        print('%r generated an exception: %s' % (url, future.exception()))
>    else:
>        print('%r page is %d bytes' % (url, len(future.result())))
>
> What do you think? Are we moving in the right direction?
>
> Cheers,
> Brian
>
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>



-- 
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/