persistent deque (continued)

[M.-A. Lemburg]

On 2008-07-21 21:08, castironpi wrote:
> Some time ago, I was asking about the feasibility of a persistent
> deque, a double-ended queue.

You might want to have a look at mxBeeBase:

http://www.egenix.com/products/python/mxBase/mxBeeBase/

Using the integer index you could probably write an on-disk
dequeue.

The B*Tree indexes available in mxBeeBase keep the indexes sorted,
so you'd only have to keep incrementing the index as you add new
values and keep a record of the highest and lowest index currently
in use.

Details are left as exercise for the interested reader ;-)

> It runs into the typical space allocation problems.  If you're storing
> a pickle, you have to allocate and fragment the file you've opened,
> since pickles can be variable-length strings; i.e. if the new data is
> too long, blank out its entry, and grow the file.  If you're storing a
> data-type, you lose Python's dynamic-type advantages, as well as its
> long integers, as they can be any length.  If you change the object in
> the deque, such as when using any mutable type, you have to update the
> container too.
> 
> Does anyone have any experience storing pickles (I am aware of the
> similarities to shelf) to a database?
> Can the file system facilitate the variable-length string problem?
> How feasible is a length-of-length - length - data solution to the
> unbounded integer problem?
> Is there any alternative to completely re-pickling a large (say 1k
> pickled) object you only change slightly?
> What other issues are there?
> Is a hybrid-storage type possible, that stores the contents of its
> Python-allocated memory block to disk, including reference count, even
> if it's a lot slower?  The object could not contain any references to
> objects not allocated on disk.
> 
> A first approach is for the file to look like this:
> 
> 00 data 01 data 02
> 01 data 03 data 04
> 02 data 05 data 06
> 
> Append would add:
> 
> 03 data 07 data 08
> 
> AppendLeft would add:
> 
> -01 data 09 data 0a
> 
> Pop would remove 03, PopLeft would remove -01.  You would need a
> length-and-head index to make 'rotate' available.  Remove would run a
> worst-case risk of renumbering half of the indices stored, plus a
> rotate.
> --
> http://mail.python.org/mailman/listinfo/python-list

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 21 2008)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611