Real-world use of concurrent.futures

[Andrew McLean]

On 08/05/2014 21:44, Ian Kelly wrote:
> I don't think it needs to be "messy". Something like this should do
> the trick, I think:
>
> from concurrent.futures import *
> from itertools import islice
>
> def batched_pool_runner(f, iterable, pool, batch_size):
>   it = iter(iterable)
>   # Submit the first batch of tasks.
>   futures = set(pool.submit(f, x) for x in islice(it, batch_size))
>   while futures:
>     done, futures = wait(futures, return_when=FIRST_COMPLETED)
>     # Replenish submitted tasks up to the number that completed.
>     futures.update(pool.submit(f, x) for x in islice(it, len(done)))
>     yield from done
>

Thank you, that's very neat. It's just the sort of thing I was looking
for. Nice use of itertools.islice and "yield from".

I'll try this out in the next few days and report back.

- Andrew