[issue42935] Pickle can't import builtins at exit

[Pete Wicken]

New submission from Pete Wicken <petewicken at gmail.com>:

Originally found as an issue in Lib/shelve.py; if we attempt to pickle a builtin as the program is exiting then Modules/_pickle.c will fail at the point of the PyImport_Import in save_global. In CPython3.8 this causes a segfault, in CPython3.9 a PicklingError is raised.

This is especially problematic in shelve.py as object pickling is attempted by the __del__ method's call stack when writeback=True. Therefore if the program exits before an explicit sync is called; in 3.8 the data will not be written to disk and a segfault occurs; in 3.9 the data is written to disk, but with an uncaught exception being raised.


Exception demonstrated via shelve on 3.9.1 on MacOS with Clang:

Python 3.9.1 (default, Dec 10 2020, 11:11:14)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> s = shelve.open('testing', writeback=True)
>>> s['a'] = Exception
>>> exit()
Exception ignored in: <function Shelf.__del__ at 0x10ed67790>
Traceback (most recent call last):
  File ".../3.9/lib/python3.9/shelve.py", line 162, in __del__
  File ".../3.9/lib/python3.9/shelve.py", line 144, in close
  File ".../3.9/lib/python3.9/shelve.py", line 168, in sync
  File ".../3.9/lib/python3.9/shelve.py", line 124, in __setitem__
_pickle.PicklingError: Can't pickle <class 'Exception'>: import of module 'builtins' failed


Segfault demonstrated via shelve on 3.8.5 on MacOS with Clang (different system from above):

Python 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> s = shelve.open('testing', writeback=True)
>>> s['a'] = Exception
>>> exit()
[1]    10040 segmentation fault  python3.8


Exception demonstrated via shelve on 3.9.1 on RHEL with GCC:

Python 3.9.1 (default, Dec  8 2020, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> s = shelve.open("thing", writeback=True)
>>> s['a'] = Exception
>>> 
Exception ignored in: <function Shelf.__del__ at 0x7f3f30b7c820>
Traceback (most recent call last):
  File "/usr/lib64/python3.9/shelve.py", line 162, in __del__
  File "/usr/lib64/python3.9/shelve.py", line 144, in close
  File "/usr/lib64/python3.9/shelve.py", line 168, in sync
  File "/usr/lib64/python3.9/shelve.py", line 124, in __setitem__
_pickle.PicklingError: Can't pickle <class 'Exception'>: import of module 'builtins' failed


Code example to reproduce using Pickle in the class __del__, demonstrated on a RHEL system:

Python 3.9.1 (default, Dec  8 2020, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pickle import DEFAULT_PROTOCOL, Pickler
>>> from io import BytesIO
>>> class T:
...   def __del__(self):
...     f = BytesIO()
...     p = Pickler(f, DEFAULT_PROTOCOL)
...     p.dump(sum)
... 
>>> t = T()
>>> exit()
Exception ignored in: <function T.__del__ at 0x7f5f04d9ef70>
Traceback (most recent call last):
  File "<stdin>", line 5, in __del__
_pickle.PicklingError: Can't pickle <built-in function sum>: import of module 'builtins' failed


Have not tested on 3.6, 3.7 or 3.10.

----------
components: Library (Lib)
messages: 385110
nosy: Wicken
priority: normal
severity: normal
status: open
title: Pickle can't import builtins at exit
type: crash
versions: Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42935>
_______________________________________