[New-bugs-announce] [issue39362] add option to make chunksize adaptive for multiprocessing.pool methods

[Forest]

New submission from Forest <fgregg at gmail.com>:

In the multiprocessing Pool methods like map, chunksize determines the trade-off between computation per task and inter-process communication. Setting chunksize appropriately has a large effect on efficiency.


However, for users directly interacting with the map methods, the way to find the appropriate chunksize is by manually checking different sizes and observing the program behavior.

For library developers, you have to hope that you set an reasonable value that will work okay across different hardware, operating systems, and task characteristics.

Generally, users of these methods want maximum throughput. It would be great if the map-like methods could adapt their chunksize towards that goal.

Something along the lines of this:

    n_items = 0
    queue = Queue(N)
    while True:
        chunk = tuple(itertools.islice(iterable, chunk_size))
        if chunk:
            queue.put(chunk)

            n_items += chunk_size
            i += 1

            if i % 10:
                time_delta = max(time.perf_counter() - t0, 0.0001)

                current_rate = n_items / time_delta

                # chunk_size is always either growing or shrinking, if
                # the shrinking led to a faster rate, keep
                # shrinking. Same with growing. If the rate decreased,
                # reverse directions
                if current_rate < last_rate:
                    multiplier = 1 / multiplier

                chunk_size = int(min(max(chunk_size * multiplier, 1), upper_bound))

                last_rate = current_rate
                n_items = 0
                t0 = time.perf_counter()


Would such a feature be desirable?

----------
components: macOS
messages: 360126
nosy: fgregg, ned.deily, ronaldoussoren
priority: normal
severity: normal
status: open
title: add option to make chunksize adaptive for multiprocessing.pool methods

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39362>
_______________________________________