dynamic allocation file buffer

[Francesc]

On 15 Set, 22:09, "Aaron \"Castironpi\" Brady" <castiro... at gmail.com>
wrote:
> On Sep 15, 4:34 am, Francesc <fal... at gmail.com> wrote:
>
>
>
> > On 12 Set, 14:39, "Aaron \"Castironpi\" Brady" <castiro... at gmail.com>
> > wrote:
>
> > > > A consideration of other storage formats such as HDF5 might
> > > > be appropriate:
>
> > > >http://hdf.ncsa.uiuc.edu/HDF5/whatishdf5.html
>
> > > > There are, of course, HDF5 tools available for Python.
>
> > > PyTablescame up within the past few weeks on the list.
>
> > > "When the file is created, the metadata in the object tree is updated
> > > in memory while the actual data is saved to disk. When you close the
> > > file the object tree is no longer available. However, when you reopen
> > > this file the object tree will be reconstructed in memory from the
> > > metadata on disk...."
>
> > > This is different from what I had in mind, but the extremity depends
> > > on how slow the 'reconstructed in memory' step is.  (Fromhttp://www.pytables.org/docs/manual/ch01.html#id2506782).  The
> > > counterexample would be needing random access into multiple data
> > > files, which don't all fit in memory at once, but the maturity of the
> > > package might outweigh that.  Reconstruction will form a bottleneck
> > > anyway.
>
> > Hmm, this was a part of a documentation that needed to be updated.
> > Now, the object tree is reconstructed in a lazy way (i.e. on-demand),
> > in order to avoid the bottleneck that you mentioned.  I have corrected
> > the docs in:
>
> >http://www.pytables.org/trac/changeset/3714/trunk
>
> > Thanks for (indirectly ;-) bringing this to my attention,
>
> > Francesc
>
> Depending on how lazy the reconstruction is, would it be possible to
> modify separate tables from separate processes concurrently?

No, modification of different tables in the same file simultaneously
is not supported yet.  This is a limitation of the HDF5 library
itself.  The HDF Group said that they have plans to address this, but
this is probably a long-term task.

Francesc