persistent composites

[kindly]

On Jun 14, 3:27 pm, Aaron Brady <castiro... at gmail.com> wrote:
> Hi, please forgive the multi-posting on this general topic.
>
> Some time ago, I recommended a pursuit of keeping 'persistent
> composite' types on disk, to be read and updated at other times by
> other processes.  Databases provide this functionality, with the
> exception that field types in any given table are required to be
> uniform.  Python has no such restriction.
>
> I tried out an implementation of composite collections, specifically
> lists, sets, and dicts, using 'sqlite3' as a persistence back-end.
> It's significantly slower, but we might argue that attempting to do it
> by hand classifies as a premature optimization; it is easy to optimize
> debugged code.
>
> The essentials of the implementation are:
>   - each 'object' gets its own table.
>     = this includes immutable types
>   - a reference count table
>     = when an object's ref. count reaches zero, its table is dropped
>   - a type-map table
>     = maps object's table ID to a string of its type
>   - a single 'entry point' table, with the table ID of the entry-point
> object
>     = the entry point is the only data structure available to new
> connections.  (I imagine it will usually be a list or dict.)
>
> I will be sure to kill any interest you might have by now, by
> "revealing" a snippet of code.
>
> The object creation procedure:
>
> def new_table( self, type ):
>   ''' 'type' is a string, the name of the class the object is an
> instance of '''
>   cur= self.conn.cursor( )
>   recs= cur.execute( '''SELECT max( tableid ) FROM refcounts''' )
>   rec= cur.fetchone( )
>   if rec[ 0 ] is None:
>     obid= 0
>   else:
>     obid= rec[ 0 ]+ 1
>   cur.execute( '''INSERT INTO types VALUES( ?, ? )''', ( obid,
> type ) )
>   cur.execute( '''INSERT INTO refcounts VALUES( ?, ? )''', ( obid,
> 1 ) )
>
> The increment ref. count procedure:
>
> def incref( self, obid ):
>   cur= self.conn.cursor( )
>   recs= cur.execute( '''SELECT count FROM refcounts WHERE tableid
> = ?''', ( obid, ) )
>   rec= recs.fetchone( )
>   newct= rec[ 0 ]+ 1
>   cur.execute( '''UPDATE refcounts SET count = ? WHERE tableid = ?''',
> ( newct, obid ) )
>
> The top-level structure contains these two procedures, as well as
> 'setentry', 'getentry', and 'revive' procedures.
>
> Most notably, user-defined types are possible.  The dict is merely a
> persistent dict.  'revive' checks the global namespace by name for the
> original type, subject to the same restrictions that we all know and
> love that 'pickle' has.
>
> As usual, deadlocks and cyclic garbage pose the usual problems.  The
> approach I used last time was to maintain a graph of acquired locks,
> and merely check for cycles to avert deadlocks, which would go in a
> separate table.  For garbage, I can't find a better solution than
> Python already uses.
>
> From the 3.0 docs:
> gc.garbage
>
>     A list of objects which the collector found to be unreachable but
> could not be freed (uncollectable objects).
> ...
> Python doesn’t collect such [garbage] cycles automatically because, in
> general, it isn’t possible for Python to guess a safe order in which
> to run the __del__() methods. If you know a safe order, you can force
> the issue by examining the garbage list, and explicitly breaking
> cycles due to your objects within the list.
>
> Before I go and flesh out the entire interfaces for the provided
> types, does anyone have a use for it?

I like it as a concept, have not got any use for it this minute, but I
am sure it will be useful someday.