[issue22005] datetime.__setstate__ fails decoding python2 pickle

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

There are two issues here.

1. datetime.datetime accepts only bytes, not str.
2. Unpickling non-ASCII str pickled in Python 2 raises an error by default.

The second issue usually hides the first one. The demonstration of the first issue:

>>> pickle.loads(b'cdatetime\ndatetime\n(U\n\x07l\x01\x01\x00\x00\x00\x00\x00\x00tR.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: an integer is required (got type str)

The first issue can be solved by accepting str argument and encoding it to bytes. The second issue can be solved by changing an encoding or an error handler. Following patch uses the "surrogateescape" error handler.

>>> pickle.loads(b'cdatetime\ndatetime\n(U\n\x07l\x01\x01\x00\x00\x00\x00\xc3\xa4tR.')
datetime.datetime(1900, 1, 1, 0, 0, 0, 50084)

Unfortunately setting the "surrogateescape" error handler by default has a side effect. It can hide string decoding errors. In addition, unpickling datetime will not always work with different encodings.

>>> pickle.loads(b'cdatetime\ndatetime\n(U\n\x07l\x01\x01\x00\x00\x00\x00\xc3\xa4tR.', encoding='latin1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-9: ordinal not in range(128)
>>> pickle.loads(b'cdatetime\ndatetime\n(U\n\x07l\x01\x01\x00\x00\x00\x00\xc3\xa4tR.', encoding='utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: an integer is required (got type str)

----------
keywords: +patch
nosy: +serhiy.storchaka
versions: +Python 3.5, Python 3.6
Added file: http://bugs.python.org/file40758/unpickle_datetime_surrogateescape.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22005>
_______________________________________