anydbm is slow

[Skip Montanaro]

    Amir> I'm trying to write out a large database file using anydbm.
    Amir> However, it's quite slow.  Writing out a 200 meg file can take
    Amir> hours.

    Amir> In fact, I do most of the work using an internal map and write out
    Amir> that map at the end since I thought this would be faster.  (I have
    Amir> lots of memory.) However, it's still very slow.

    Amir> Any hints?

What underlying db module does anydbm use?

    >>> import anydbm
    >>> anydbm._defaultmod 
    <module 'dbhash' from '/usr/local/lib/python2.2/dbhash.pyc'>

If it returns the dumbdbm module, you'll have a good idea where the problem
lies.  On the other hand, writing out a 200meg file is going to take awhile.
The underlying db package probably has to do quite a bit of shuffling.

If you're accessing a relatively small number of entries in the db file,
you'd probably be better off letting the underlying library cache things for
you.

-- 
Skip Montanaro (skip at pobox.com - http://www.mojam.com/)