[Numpy-discussion] [SciPy-User] recommendation for saving data

[Anne Archibald]

In astronomy we tend to use FITS, which is well-supported by pyfits,
but a little limited. Some new instruments are beginning to use HDF5.

All these generic formats allow very general data storage, so you will
need to come up with a standrdized way to represent your own data.
Used well, these formats can be self-describing enough that generic
tools can be very useful (e.g. display images, build histograms) but
it takes some thought when designing files.

Anne

On 8/1/11, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> On 7/31/11 5:48 AM, Brian Blais wrote:
>> I was wondering if there are any recommendations for formats for saving
>> scientific data.
>
> every field has it's own standards -- I'd try to find one that is likely
> to be used by folks that may care about your results.
>
> For Oceanographic and Atmospheric modeling data, netcdf is a good
> option. I like the NetCDF4 python lib:
>
> http://code.google.com/p/netcdf4-python/
>
> (there are others)
>
> For broader use, and a bit more flexibility, HDF is a good option. There
> are at least two ways to use it with numpy:
>
> PyTables: http://www.pytables.org
>
> (Nice higher-level interface)
>
> hf5py:
> http://alfven.org/wp/hdf5-for-python/
>
> (a more raw HDF5 wrapper)
>
> There is also the npz format, built in to numpy, if you are happy with
> requiring python to read the data.
>
> -Chris
>
>
>   I am running a simulation, which has many somewhat-indepedent parts
> which have their own internal state and parameters.  I've been using
> pickle (gzipped) to save the entire object (which contains subobjects,
> etc...), but it is getting too unwieldy and I think it is time to look
> for a more robust solution.  Ideally I'd like to have something where I
> can call a save method on the simulation object, and it will call the
> save methods on all the children, on down the line all saving into one
> file.  It'd also be nice if it were cross-platform, and I could depend
> on the files being readable into the future for a while.
>>
>> Are there any good standards for this?  What do you use for saving
>> scientific data?
>>
>>
>> 		thank you,
>>
>> 			Brian Blais
>>
>>
>>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

-- 
Sent from my mobile device