[Numpy-discussion] py2/py3 pickling

[Chris Laumann]

Hi all-

Is there documentation about the limits and workarounds for py2/py3 pickle/np.save/load compatibility? I haven't found anything except developer bug tracking discussions (eg. #4879 in github numpy).

The kinds of errors you get can be really obscure when save/loading complicated objects or pickles containing numpy scalars. It's really unclear to me why the following shouldn't work -- it doesn't have anything apparent to do with string handling and unicode.

Run in py2:

import pickle
import numpy as np

a = np.float64(0.99)
pickle.dump(a, open('test.pkl', 'wb'))

And then in py3:

import pickle
import numpy as np

b = pickle.load(open('test.pkl', 'rb'))

And you get:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xae in position 0: ordinal not in range(128)

If you force encoding='bytes' in the load, it works.

Is this explained anywhere?

Best, C