[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Mark Dickinson]

Mark Dickinson added the comment:

A new patch (with tests), and a fuller explanation:

At work, we've got Python talking to a customer's existing COM library; we're using Thomas Heller's 'comtypes' library to do that.  Unfortunately, comtypes depends quite a lot on __del__-time cleanup, so reference counting matters.  (I'm well aware that this isn't the recommended way to deal with resource cleanup in Python, but rewriting the existing infrastructure isn't a realistic option here.)

Anyway, it turned out that the concurrent.futures executors were indirectly holding onto references to COM objects, causing issues with our application.

The attached patch adds a few 'del' statements to remove references that are no longer needed.  For the ProcessExecutor, some of those 'del' statements had to go into the multiprocessing.Queue implementation.

The troublesome pattern (in both multiprocessing and futures) takes the form (simplified):


def my_worker_function(...):
    ...
    while <exit_condition_not_satisfied>:
        obj = blocking_wait_for_next_item()
        do_processing(obj)
    ...

The issue is that the reference to obj is kept until the completion of the next blocking wait call.  I'm suggesting just adding an extra 'del obj' after 'do_processing(obj)'.

----------
components: +Library (Lib)
Added file: http://bugs.python.org/file27633/kill_reference_2.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16284>
_______________________________________