[New-bugs-announce] [issue31993] pickle.dump allocates unnecessary temporary bytes / str

[Olivier Grisel]

New submission from Olivier Grisel <olivier.grisel at ensta.org>:

I noticed that both pickle.Pickler (C version) and pickle._Pickler (Python version) make unnecessary memory copies when dumping large str, bytes and bytearray objects.

This is caused by unnecessary concatenation of the opcode and size header with the large bytes payload prior to calling self.write.

For protocol 4, an additional copy is caused by the framing mechanism.

I will submit a pull request to fix the issue for the Python version. I am not sure how to test this properly. The BigmemPickleTests seems to be skipped on my 16 GB laptop.

----------
components: Library (Lib)
messages: 305975
nosy: Olivier.Grisel, pitrou
priority: normal
severity: normal
status: open
title: pickle.dump allocates unnecessary temporary bytes / str
type: resource usage
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31993>
_______________________________________