PEP 285 and pickle compatibility

[Guido van Rossum]

Paul Rubin wrote:
> Proposed fix: in new pickles, dump boolean True as "I01" and boolean
> False as "I00".  The current unpickler should load these as the integer
> values 1 and 0 and everything should work fine.  But these strings
> will never be generated by the current pickler, which wouldn't emit
> the leading 0's.
> 
> In the new unpickler, put a special hack into the load_int routines
> in cPickle.c and pickle.py, that notice the leading 0 and push the
> appropriate bool value.

I like this.  I'll see if I can implement it.  (Though it's not clear what
to do in binary mode.)

> I think the ugliness of this kludge is outweighed by escaping making
> an incompatible change to the pickle format.  Sooner or later, though,
> a real incompatible change will be needed, so I'd like to propose
> a further change now, that shouldn't cause compatibility probs:
> 
> The current pickle dump/dumps functions currently take a "bin" arg
> specifying whether to dump in text or binary format.  Unfortunately,
> this arg is treated as a bool and the dumps routine only cares whether
> it's true or false.  So I think a further optional arg should be
> added, giving a "format code" saying what unpickler versions have
> to be able to read the pickle being created:
> 
>   dump(object, format=0)  # same as dump(object), text format pickle
>   dump(object, format=1)  # same as dump(object,bin=1), binary format
>   dump(object, format=2)  # new, incompatible format
> 
> Pickles created with format=2 might for example actually contain a
> bool code instead of the I00/I01 hack, so old unpicklers couldn't read them.
> 
> If a future pickler/unpickler introduces an incompatible format, one
> feature I'd hope to see added is efficient binary pickling of long
> ints.  Currently they're pickled as ascii decimal strings (even in
> binary pickles), which are very slow to convert to and from (all that
> arithmetic).

I like this recommendation, but I'm not motivated enough to implement
it myself.  Maybe you can get some people to work with you on a PEP
and a sample implementation?

--Guido van Rossum (home page: http://www.python.org/~guido/)