[SciPy-User] HDF4, HDF5, netcdf solutions -- PyNIO/PyNGL or CDAT or ??

[Zachary Pincus]

Ack -- I didn't mean to set off a big back and forth! I'd only wanted  
to convey that h5py and pytables seem to serve different purposes: one  
a "simple and thin" pythonic wrapper around the official hdf5  
libraries, and the other seeking to provide some value-add on top of  
that.

I guess I got something of the rationale for using h5py over pytables  
wrong -- but there is some rationale, right? What is that?


On Nov 1, 2010, at 6:02 PM, Francesc Alted wrote:

> A Monday 01 November 2010 21:19:02 Robert Kern escrigué:
>> On Mon, Nov 1, 2010 at 15:01, Dav Clark <dav at alum.mit.edu> wrote:
>>> On Nov 1, 2010, at 5:14 AM, Zachary Pincus wrote:
>>>> Where pytables tries to present its own interface, h5py just gives
>>>> you the hdf5 file. This means that pytables can do a lot of neat
>>>> things (like the indexed searching), but it also means that (at
>>>> least last I checked) pytables isn't the best tool for reading in
>>>> hdf5 files not created by pytables -- for that, you'd want h5py.
>>>
>>> Every time I've had an issue with pytables reading a non-pytables
>>> created file, I've submitted a bug and it got fixed usually in a
>>> few days. At the time, I was using HDF5 as a transfer layer
>>> between matlab's rudimentary hdf5 support and python w/ pytables.
>>> (Thanks Francesc!)
>>
>> I just wanted to add that in my experience, you can read just about
>> any HDF5 file with PyTables except for a few with some more exotic
>> features.
>
> Let me chime in just to try to clarify couple of things.  First, both
> PyTables and h5py can read most of the HDF5 files out there, but  
> none of
> them has *complete* support for HDF5 files (implementing complete
> support for the whole HDF5 standard is really a tough task).  In
> addition, the last time that I checked this (about one year ago, so
> things might have changed since then), PyTables can read (and create)
> HDF5 files that h5py cannot; and the contrary is true too.
>
>> If you absolutely need to write an HDF5 file according to a
>> strict standard without any extra bits, you may need h5py. However,
>> many other readers of your standard probably won't care about the
>> extra bits PyTables includes.
>
> I suppose that the 'extra bits' you are referring to are the HDF5
> attributes that complement HDF5 nodes as metainfo.  Let me say that  
> most
> of these attributes are not PyTables-specific, but those used in the
> high-level API of HDF5 (http://www.hdfgroup.org/HDF5/doc/HL/).   
> Anyway,
> as I said many times, if these attributes are causing some trouble to
> the user (they should not), you can always disable its creation by
> setting the PYTABLES_SYS_ATTRS parameter to false during the opening  
> of
> a file (or, if you like this to be permanent, in the
> tables/parameters.py).  For more info about this, see:
>
> http://www.pytables.org/docs/manual/apc.html#id364726
>
>> You just have to be a little bit
>> careful to make sure that you aren't relying on any PyTables
>> features, like Python-pickled attributes.
>
> PyTables only uses pickle when trying to save attributes that are not
> supported by HDF5 (with the exception of unicode strings that should  
> be
> implemented soon in PyTables).  For example, if you try to save a list
> as an attribute:
>
> node.attrs.my_attr = [1,2,[3,4]]
>
> as such a list cannot be represented by HDF5 natively, PyTables  
> chooses
> to pickle it and save it.  During retrieval, the pickle is  
> automatically
> detected and unpickled before being returned to the user.  Of course,
> you will not be able to read such attributes with a non-Python
> application.  And, although I consider this like a feature, I can
> understand that this might be considered as a bug by others (but I  
> have
> to say that very few PyTables users, if any at all, has ever  
> complained
> about this 'feature'/'bug').
>
> Hope this helps clarifying some points,
>
> -- 
> Francesc Alted
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user