[issue39681] pickle.load expects an object that implements readinto

[Nathan Goldbaum]

Nathan Goldbaum <nathan12343 at gmail.com> added the comment:

So I *think* I've pieced together what caused the user crash that originated in the flair library. It turns out that pickle.load, via torch.load, is getting passed an mmap.mmap. 

https://github.com/flairNLP/flair/blob/1d44abf35f1122d1e146c9a219b2cb5f98b41b40/flair/file_utils.py#L25-L36

https://github.com/flairNLP/flair/blob/1d44abf35f1122d1e146c9a219b2cb5f98b41b40/flair/nn.py#L85-L86

Since mmap doesn't implement readinto, pickle.load objects as of Python 3.8. This is new behavior in Python3.8, it used to be possible to load a memory-mapped pickle file.

Short repro script:

import pickle
import mmap
data = "some data"

with open('my_data.pkl', 'wb') as f:
    pickle.dump(data, f)

with open("my_data.pkl", "r+b") as f_in:
    mm = mmap.mmap(f_in.fileno(), 0)

print(pickle.load(mm))

On Python3.8, this script prints an error, on Python3.7 it prints "some data".

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39681>
_______________________________________