[New-bugs-announce] [issue34572] C unpickling bypasses import thread safety

[Tim Burgess]

New submission from Tim Burgess <ozburgess at gmail.com>:

Retrieving and using a module directly from sys.modules (from C in this case) leads to a race condition where the module may be importing on another thread but has not yet been initialised.  For slow filesystems or large modules (e.g. numpy) this seems to lead to easily reproducible errors (the attached code fails 100% of the time on my work machine - CentOS 7).

I believe they have to be in sys.modules during this phase due to the possibility of circular references.

importlib handles this carefully with locking, but _pickle.c bypasses all that, leading to issues with threaded codes that use pickling, e.g. dask/distributed.

----------
components: Extension Modules
files: reproducer_submit.py
messages: 324528
nosy: tjb900
priority: normal
severity: normal
status: open
title: C unpickling bypasses import thread safety
type: crash
versions: Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file47784/reproducer_submit.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34572>
_______________________________________