[SciPy-user] Fast saving/loading of huge matrices

[Pauli Virtanen]

Sat, 21 Apr 2007 10:53:50 +0200, Gael Varoquaux kirjoitti:
> On Fri, Apr 20, 2007 at 04:43:36PM +0000, Pauli Virtanen wrote:
>> In a different attempt to make storing stuff in Pytables easier, I
>> wrote a library to dump and load any objects directly to HDF5 files
> 
>> 	http://www.iki.fi/pav/software/hdf5pickle/index.html
> 
> Do you think this can be used to save data in a way that can be used to
> share it between programs ? Something a bit more unniversal than
> python's pickle.
> If so I vote for inclusion in pytables.

I guess that between different Python programs both using hdf5pickle, the
sharing characteristics are the same as for Python pickle: you can share
objects if their class is present in both programs.

Sharing data between a Python program A using hdf5pickle and a (possibly
non-Python) program B not using it works at least in the direction A -> B
for the data that can be saved in a native HDF5 format (e.g. ints, floats,
arrays, dicts, __dicts__ of objects etc). Direction B->A is more tricky,
as hdf5pickle currently expects to find a 'pickletype' attribute
describing what type of an object is stored in a node. A simple fallback
should be easy to implement, groups to dicts and others to arrays.

-- 
Pauli