[Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data

[Chris Jerdonek]

On Wed, Mar 28, 2018 at 6:15 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Wed, Mar 28, 2018 at 1:03 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 28.03.18 21:39, Antoine Pitrou пише:
>>> I'd like to submit this PEP for discussion.  It is quite specialized
>>> and the main target audience of the proposed changes is
>>> users and authors of applications/libraries transferring large amounts
>>> of data (read: the scientific computing & data science ecosystems).
>>
>> Currently I'm working on porting some features from cloudpickle to the
>> stdlib. For these of them which can't or shouldn't be implemented in the
>> general purpose library (like serializing local functions by serializing
>> their code objects, because it is not portable) I want to add hooks that
>> would allow to implement them in cloudpickle using official API. This would
>> allow cloudpickle to utilize C implementation of the pickler and unpickler.
>
> There's obviously some tension here between pickle's use as a
> persistent storage format, and its use as a transient wire format. For
> the former, you definitely can't store code objects because there's no
> forwards- or backwards-compatibility guarantee for bytecode. But for
> the latter, transmitting bytecode is totally fine, because all you
> care about is whether it can be decoded once, right now, by some peer
> process whose python version you can control -- that's why cloudpickle
> exists.

Is it really true you'll always be able to control the Python version
on the other side? Even if they're internal services, it seems like
there could be times / reasons preventing you from upgrading the
environment of all of your services at the same rate. Or did you mean
to say "often" all you care about ...?

--Chris



>
> Would it make sense to have a special pickle version that the
> transient wire format users could opt into, that only promises
> compatibility within a given 3.X release cycle? Like version=-2 or
> version=pickle.NONPORTABLE or something?
>
> (This is orthogonal to Antoine's PEP.)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com