[Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data

[Nathaniel Smith]

On Wed, Mar 28, 2018 at 1:03 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 28.03.18 21:39, Antoine Pitrou пише:
>> I'd like to submit this PEP for discussion.  It is quite specialized
>> and the main target audience of the proposed changes is
>> users and authors of applications/libraries transferring large amounts
>> of data (read: the scientific computing & data science ecosystems).
>
> Currently I'm working on porting some features from cloudpickle to the
> stdlib. For these of them which can't or shouldn't be implemented in the
> general purpose library (like serializing local functions by serializing
> their code objects, because it is not portable) I want to add hooks that
> would allow to implement them in cloudpickle using official API. This would
> allow cloudpickle to utilize C implementation of the pickler and unpickler.

There's obviously some tension here between pickle's use as a
persistent storage format, and its use as a transient wire format. For
the former, you definitely can't store code objects because there's no
forwards- or backwards-compatibility guarantee for bytecode. But for
the latter, transmitting bytecode is totally fine, because all you
care about is whether it can be decoded once, right now, by some peer
process whose python version you can control -- that's why cloudpickle
exists.

Would it make sense to have a special pickle version that the
transient wire format users could opt into, that only promises
compatibility within a given 3.X release cycle? Like version=-2 or
version=pickle.NONPORTABLE or something?

(This is orthogonal to Antoine's PEP.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org