[New-bugs-announce] [issue21345] multiprocessing.Pool._handle_workers sleeps too long

[Johannes Baiter]

New submission from Johannes Baiter:

While testing a module that uses multiprocessing.Pool to distribute load across multiple processes, I noticed that my test suite was copmleting very quickly (~0.15s) on Python 2.6, while Python 2.7 and above took around 10x as long (~1.6s).
Upon debugging this, I pinned the slowdown down to the 'Pool.join()' method. Removing it removed the slowdown almost completely.
I then checked the version history of the 'multiprocessing.pool' module between 2.6 and 2.7 and noticed that when the 'maxtasksperchild' parameter was introduced, a thread to handle the workers was introduced, which was 'join()'ed when the pool was joined.

This is the function that is executed in the thread (from latest CPython checkout):

    @staticmethod
    def _handle_workers(pool):
        thread = threading.current_thread()

        # Keep maintaining workers until the cache gets drained, unless the pool
        # is terminated.
        while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
            pool._maintain_pool()
            time.sleep(0.1)  #  <-- Cause of slow 'join()' after 2.6
        # send sentinel to stop workers
        pool._taskqueue.put(None)
        util.debug('worker handler exiting')

I highlighted the portion that makes 'join()' take a rather long time with short-lived processes in Python 2.7 and greater.
Replacing it with 'time.sleep(0)' (as is done in '_help_stuff_finish()' later in the module) makes joining go as fast as in 2.6.

Is there a specific reason why this sleep period was chosen?

----------
components: Library (Lib)
messages: 217129
nosy: Johannes.Baiter
priority: normal
severity: normal
status: open
title: multiprocessing.Pool._handle_workers sleeps too long
type: performance
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21345>
_______________________________________