Pickled objects over the network

[Hendrik van Rooyen]

"Steve Holden" <steve at ho...eb.com> wrote:

> I think someone has already pointed out netstrings, which will allow you 
> to send arbitrary strings over network connections deterministically. 

Yes I brought it up

> I'm afraid for the rest it's just a matter of encoding your information 
> in a way that you can decode without allowing a malicious sender to 
> cause arbitrary code to be called.

Yes - and in general you do this by having both the sender and the 
transmitter conform to some pre agreed format - a netstring is one 
of the simplest of such things - another way is to "frame" records 
between some kind of delimeter and to "escape" the occurences of the
delimiter in the data.  Another way is to use simple "self defining fields"
that work by giving fields a "tag" number from a list of pre defined
things, as well as a length, followed by the data - some financial 
protocols work as a variant of this concept, where the presence or 
absence of a bit signify the presence or absence of a field in the record.

The problem with all of these schemes is that they are all a PITA to
implement, compared to the ease with which you can pickle and 
unpickle something like a simple dict of parameters.

And if that is all you want to pass over to some remote thing, then
having to download and import Pyro is an equal PITA and overkill.
- It adresses a far more sophisticated problem than just getting 
some small things across the network.

Now if Pyro were to make it into the standard library, it would be
my method of choice for even this silly level of functionality, 
because I happen to think it rocks.

> 
> The issue with pickle is that it's way too general a mechanism to be 
> secure in open network applications, so a suggestion to beef up its 
> security was misguided. Trying to "beef up pickle's security" is like 
> trying to make a shotgun that can't kill anything.
> 

Is it really that impossible to add something like a "noeval" flag, or to
force it to only give you a string or a dict if you ask for one or the other, 
given that someone has already mentioned that the built in types are 
decoded by separate routines?

Or more generally - as it already has different protocols - to define a
protocol that won't pass executable stuff over, or one that will only 
pass and accept the built in types?

- Hendrik