[SciPy-User] Custom array serialization

[Jeff Reback]

Pandas 0.12 (releasing shorty), will have full-dtype support for
JSON serialization/deserialization of DataFrames via a bundled USJON parser

see here: http://pandas.pydata.org/pandas-docs/dev/io.html#json


On Jul 3, 2013, at 5:21 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Wed, Jul 3, 2013 at 4:54 AM, Chris Weisiger <cweisiger at msg.ucsf.edu> wrote:
> >
> > I'm working on a game project; more specifically, right now I'm working on saving and loading the game. As a result, I need to serialize the game state to a file, and deserialize it later.
> >
> > To pre-empt some responses, I spent a lot of time thinking about this before starting, and came to the conclusion that pickle and other similar automatic [de]serialization libraries were not suitable for this problem. The sticking point is that these libraries invariably let you put code into the serialized object, which code is then executed when you deserialize it. As a result, if you have the deserialization routine in your code, then you have a security breach. I would rather my users be able to distribute savefiles without worrying that one of them has been sabotaged to do something malicious.
> >
> > Instead, I'm manually serializing to JSON, and manually deserializing. It's actually working decently well so far. I've hit one minor sticking point though: numpy array serialization. Of course I'm aware of numpy.tostring(), but that doesn't preserve type information. And I don't know of a good way to serialize the type and then deserialize it later.
> 
> Use the .npy format that np.save() uses:
> 
>   https://github.com/numpy/numpy/blob/master/numpy/lib/format.py
> 
> The write_array() and read_array() functions are the ones you would use.
> 
> --
> Robert Kern
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20130703/79920a00/attachment.html>