[Python-Dev] PEP 574 (pickle 5) implementation and backport available

[Neil Schemenauer]

On 2018-05-25, Antoine Pitrou wrote:
> The question is what purpose does it serve for pickle to do it rather
> than for the user to compress the pickle themselves.  You're basically
> saving one line of code.

It's one line of code everywhere pickling or unpicking happens.  And
you probably need to import a compression module, so at least two
lines.  Then maybe you need to figure out if the pickle is
compressed and what kind of compression is used.  So, add a few more
lines.

It seems logical to me that users of pickle want it to be fast and
produce small pickles.  Compressing by default seems the right
choice, even though it complicates the implementation.  Ivan brings
up a valid point that compressed pickles are harder to debug.
However, I think that's much less important than being small.

> it requires us to ship the lz4 library with Python

Yeah, that's not so great.  I think zlib with Z_BEST_SPEED would be
fine.  However, some people might worry it is too slow or doesn't
compress enough.  Having lz4 as a battery included seems like a good
idea anyhow.  I understand that it is pretty well established as a
useful compression method.  Obviously requiring a new C library to
be included expands the effort of implementation a lot.

This discussion can easily lead into bikeshedding (e.g. relative
merits of different compression schemes).  Since I'm not
volunteering to implement anything, I will stop responding at this
point. ;-)

Regards,

  Neil