[Python-ideas] Adding a safe alternative to pickle in the standard library

[Steve Dower]

> From: Stephen J. Turnbull [mailto:stephen at xemacs.org]
> Sent: Thursday, February 21, 2013 1030
> To: Steve Dower
> Cc: Andrew Barnert; Devin Jeanpierre; Dustin J. Mitchell; Eric V. Smith;
> Python-Ideas
> Subject: Re: [Python-ideas] Adding a safe alternative to pickle in the standard
> library
> 
> Steve Dower writes:
> 
>  > In a past life I used pickle regularly to snapshot long-running  >
> (evolutionary) algorithms that used user-provided classes
> 
> And how do you propose to prevent user-provided exploits, then?
>
> Nobody wants to take away the power of pickle if it imposes only risks you're
> happy to bear.  The question here is "is it possible to be
> *safer* than pickle without giving up any of the power?"
> 

I was only aiming to provide perspective, rather than a proposal. The existing ability to customise the unpickler suited my needs perfectly, not that I actually used it. To make it safer I would have a restricted unpickler as the default (for load/loads) and force people to override it if they want to loosen restrictions.

To be really explicit, I would make load/loads only work with built-in types. For compatibility when reading earlier protocols we could add a type representing a class instance and its members that doesn't actually construct anything. (Maybe we override __getattr__ to make it lazily construct the instance when the program actually wants to use it?)

For convenience, I'd add a parameter to Unpickler to let the user provide a set of types that are allowed to be constructed (or a mapping from names to callables as used in find_class()).

Finally, I'd give greater exposure to overriding find_class(), as has already been suggested. People who want to unpickle any arbitrary type have to do this to opt-in. (Maybe that's a little aggressive?)

I'd expect this would come under a new protocol version, but I wouldn't be opposed to old data being unpickled under these rules, especially if the non-constructing type is actually a lazy-constructing type.


Cheers,
Steve