[issue22005] datetime.__setstate__ fails decoding python2 pickle

[Christian Tanzer]

Christian Tanzer added the comment:

> The code works when using encoding='bytes'. Thanks Tim for the suggestion.

> So this is not a bug, but is there any sense in having encoding='ASCII' by default in pickle ?

It is most definitely a bug. And it adds another road block to moving python applications from 2.7 to 3.x!

encoding='bytes' has serious side effects and isn't useful in the general case. For instance, it will result in dict-keys being unpickled as bytes instead of as str after which hilarity ensues.

I got the exception

  UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 1: ordinal not in range(128)

when testing an application for compatibility in Python 3.5 on a pickle created by Python 2.7. The pickled data is a nested data structure and it took me quite a while to determine that the single datetime instance was the culprit.

Here is a small test case that reproduces the problem::

# -*- coding: utf-8 -*-
# pickle_dump.py 
import datetime, pickle, uuid
dti = datetime.datetime(2015, 10, 12, 13, 17, 42, 123456)
data = { "ascii" : "abc", "text" : u"äbc", "int" :  42, "date-time" : dti }
with open("/tmp/pickle.test", "wb") as file :
    pickle.dump(data, file, protocol=2)

# pickle_load.py
# -*- coding: utf-8 -*-
import pickle
with open("/tmp/pickle.test", "rb") as file :
    data = pickle.load(file)
print(data)

$ python2.7 pickle_dump.py
$ python2.7 pickle_load.py 
{'ascii': 'abc', 'text': u'\xe4bc', 'int': 42, 'date-time': datetime.datetime(2015, 10, 12, 13, 17, 42, 123456)}
$ python3.5 pickle_load.py 
Traceback (most recent call last):
  File "pickle_load.py", line 6, in <module>
    data = pickle.load(file)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 1: ordinal not in range(128)

That error message is spectacularly useless.

----------
nosy: +tanzer at swing.co.at

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22005>
_______________________________________