[New-bugs-announce] [issue37265] Memory leaks regression caused by: Fix threading._shutdown() race condition

[STINNER Victor]

New submission from STINNER Victor <vstinner at redhat.com>:

https://buildbot.python.org/all/#/builders/189/builds/60

Examples:

test_ssl leaked [3, 3, 3] references, sum=9
test_ssl leaked [3, 3, 3] memory blocks, sum=9
test_decimal leaked [4, 4, 4] references, sum=12
test_decimal leaked [4, 4, 4] memory blocks, sum=12
test_httplib leaked [1, 1, 1] references, sum=3
test_httplib leaked [1, 1, 1] memory blocks, sum=3
test_multiprocessing_fork leaked [27, 27, 27] references, sum=81
test_multiprocessing_fork leaked [27, 27, 27] memory blocks, sum=81
test_multiprocessing_forkserver leaked [27, 27, 27] references, sum=81
test_multiprocessing_forkserver leaked [27, 27, 27] memory blocks, sum=81
test_socketserver leaked [15, 15, 15] references, sum=45
test_socketserver leaked [15, 15, 15] memory blocks, sum=45
test_capi leaked [17, 17, 17] references, sum=51
test_capi leaked [17, 17, 17] memory blocks, sum=51
test_smtplib leaked [47, 47, 47] references, sum=141
test_smtplib leaked [47, 47, 47] memory blocks, sum=141
test_io leaked [23254, 23250, 23254] references, sum=69758
test_io leaked [7703, 7701, 7703] memory blocks, sum=23107
test_pickle leaked [6, 6, 6] references, sum=18
test_pickle leaked [6, 6, 6] memory blocks, sum=18
test_httpservers leaked [43, 43, 43] references, sum=129
test_httpservers leaked [43, 43, 43] memory blocks, sum=129
test_nntplib leaked [1, 1, 1] references, sum=3
test_nntplib leaked [1, 1, 1] memory blocks, sum=3
test_os leaked [5, 5, 5] references, sum=15
test_os leaked [5, 5, 5] memory blocks, sum=15
test_subprocess leaked [4, 4, 4] references, sum=12
test_subprocess leaked [4, 4, 4] memory blocks, sum=12
test_socket leaked [7, 7, 7] references, sum=21
test_socket leaked [7, 7, 7] memory blocks, sum=21
test_multiprocessing_spawn leaked [27, 27, 27] references, sum=81
test_multiprocessing_spawn leaked [27, 27, 27] memory blocks, sum=81
test_concurrent_futures leaked [3, 3, 3] references, sum=9
test_concurrent_futures leaked [3, 3, 3] memory blocks, sum=9
test_asyncio leaked [62, 62, 62] references, sum=186
test_asyncio leaked [62, 62, 62] memory blocks, sum=186


I used git bisect and I found... my own change :-)

commit 468e5fec8a2f534f1685d59da3ca4fad425c38dd
Author: Victor Stinner <vstinner at redhat.com>
Date:   Thu Jun 13 01:30:17 2019 +0200

    bpo-36402: Fix threading._shutdown() race condition (GH-13948)
    
    Fix a race condition at Python shutdown when waiting for threads.
    Wait until the Python thread state of all non-daemon threads get
    deleted (join all non-daemon threads), rather than just wait until
    Python threads complete.
    
    * Add threading._shutdown_locks: set of Thread._tstate_lock locks
      of non-daemon threads used by _shutdown() to wait until all Python
      thread states get deleted. See Thread._set_tstate_lock().
    * Add also threading._shutdown_locks_lock to protect access to
      threading._shutdown_locks.
    * Add test_finalization_shutdown() test.

----------
components: Library (Lib)
messages: 345476
nosy: vstinner
priority: normal
severity: normal
status: open
title: Memory leaks regression caused by: Fix threading._shutdown() race condition
versions: Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37265>
_______________________________________