[issue38039] Segfault when pickling dictionary with large pandas dataframes

[Ilya Valmianski]

New submission from Ilya Valmianski <ivalmian at gmail.com>:

Tried pickling a dictionary with multiple pandas tables and python primitive types. Pandas tables are large so full object size is ~200GB but system should not be OOM (crashed with ~300 GB system memory available). Reproduced on two machines running RHEL 7.5. Tried using Python 3.6 and 3.7. Tried pickle and dill. All python versions were installed as environments via Anaconda. All segfault in pickle. Here is a dump:

Fatal Python error: Segmentation fault

Current thread 0x00007f724af23740 (most recent call first):
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 496 in save
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 805 in _batch_appends
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 781 in save_list
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 476 in save
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 751 in save_tuple
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 476 in save
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 634 in save_reduce
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 521 in save
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 847 in _batch_setitems
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 821 in save_dict
  File "/apps/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 893 in save_module_dict
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 476 in save
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 634 in save_reduce
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 521 in save
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 847 in _batch_setitems
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 821 in save_dict
  File "/apps/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 893 in save_module_dict
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 476 in save
  File "/apps/anaconda3/lib/python3.6/pickle.py", line 409 in dump
  File "/apps/anaconda3/lib/python3.6/site-packages/dill/_dill.py", line 286 in dump
  File "precompute_control_patients.py", line 220 in main
  File "/apps/anaconda3/lib/python3.6/site-packages/absl/app.py", line 251 in _run_main
  File "/apps/anaconda3/lib/python3.6/site-packages/absl/app.py", line 300 in run
  File "precompute_control_patients.py", line 227 in <module>

----------
messages: 351213
nosy: Ilya Valmianski
priority: normal
severity: normal
status: open
title: Segfault when pickling dictionary with large pandas dataframes
type: crash
versions: Python 3.6, Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue38039>
_______________________________________