New release of BerkeleyDB Backend Storage Engine for DURUS (20060629)

[Jesus Cea]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

http://www.argo.es/~jcea/programacion/durus-berkeleydbstorage.htm

See the CHANGES and the UPGRADING documents.

Also updated the "KNOW_HOW-DURUS" to Durus version 3.4:
http://www.argo.es/%7Ejcea/artic/know_how-durus-3_4.htm


CHANGES:

RELEASE 20060629:

20060629 - r191 - jcea en argo.es

Document the upgrade process.

20060620 - r188 - jcea en argo.es

"get_size()" is very costly. With current implementation,
the storage must access all the database. I can implement a manual
counter or migrate to btree and use DB_RECNUM:

  http://www.sleepycat.com/docs/api_c/db_stat.html
  http://www.sleepycat.com/docs/api_c/db_set_flags.html#DB_RECNUM

Finally we implement a manually managed counter. Now "get_size()"
is instantaneous, but it require an storage upgrade. So, you can't
use previous backend versions.

20060620 - r184 - jcea en argo.es

When creating a new storage database, be able to
choose between "btree" and "hash" schema.

20060620 - r183 - jcea en argo.es

Document "KNOW_HOW-DURUS" updated to Durus 3.4.

20060524 - r157 - jcea en argo.es

We add a new "validate_garbage" optional parameter to the
storage constructor. If that parameter is True, the storage will
do a garbage check. Read the documentation in the README file
for details.

20060523 - r156 - jcea en argo.es

If the storage was stopped before garbage
collection was completed, the storage could leak
some unreachable objects.

Solving this issue, we make sure also that the garbage
collection makes progress even if the storage is stopped
repeatly before garbage collection completion. That is,
we store partial state to avoid to try to clean
already collected objects.


UPGRADING:

***
*** Release 20060629:
***

Storage databases created with this release are not
compatible with previous releases.

First time you use this release to open a storage
database created by previous releases, it will be
transparently "upgraded" to current format, so:

- - The storage will become incompatible with previous
  releases.

- - In order to be able to upgrade the storage, you can't
  open it in "read only" mode. Once upgraded, you can
  use "read only" mode freely.

- - The upgrade process can be a bit slow, since the
  backend needs to analyze the entire database. So, if
  your database size is 1 Gigabyte and your hard disk
  can push 50MB/s, the upgrade process will take about
  20 seconds to finish.

- - While upgrading, the database will NOT serve requests.

- - The upgrade process doesn't take any RAM.

- - If the upgrade process is aborted (program quits, crashes,
  machine reboot, etc), the database will be stable
  and clean. That is, the upgrade process is transactional
  and SAFE.


Also, in previous backend releases there was a bug in
the garbage collection code that would skip over
"to be deleted" objects, leaving some garbage behind.

So with this release you have three options:

- - Ignore the issue: This release will not leak new garbage,
  but garbage already leaked will remain there. You lose
  some diskspace. No other side effect.

- - Do a dump+load migration, using the storage backend
  "migrate" method. Your diskspace requirement will
  double (temporaly), since you need space for the
  source and the destination databases. The migration
  will copy garbage, but this new backend release can
  cope with it and it will be freed. You don't need
  extra RAM to do this process, even if your database
  is in the petabyte range :-).

- - Initialize the storage with the "validate_garbage"
  parameter sets to "True": This option will force
  a full sweep over the database, to examine
  cross-object links and locate garbage. This scan
  is done in-place, so you don't need extra diskspace.
  It is transactional, so it is SAFE if something
  goes wrong.

  It takes RAM proportional to object count
  in your storage, so beware if you have a huge
  database.

  Of course you only need to pass this parameter
  once, to catch garbage leaked by previous releases.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea en argo.es http://www.argo.es/~jcea/ _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea en jabber.org         _/_/    _/_/          _/_/_/_/_/
                               _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBRKQEeZlgi5GaxT1NAQKucgP/egc+jTamJRrVEVsoACG+6PKoI/pS3W72
Xh6JCeRcLdti7unql3dYNprPTbW2aNGNmOqvWa3fxGeQL02gRUge4f2l8v+Ilwzj
Hum5yP7BtOQyLUooWyZkfvsfHcw5egKQ9+tJvFlgbZk0xN5w2qb2gskW/dqYNqd+
Y9JXZ5Ipuzo=
=B8Wy
-----END PGP SIGNATURE-----