increasing the page size of a dbm store?

Tim Chase python.list at tim.thechases.com
Tue Nov 26 20:24:32 EST 2019


Working with the dbm module (using it as a cache), I've gotten the
following error at least twice now:

  HASH: Out of overflow pages.  Increase page size
  Traceback (most recent call last):
  [snip]
  File ".py", line 83, in get_data
    db[key] = data
  _dbm.error: cannot add item to database

I've read over the py3 docs on dbm

https://docs.python.org/3/library/dbm.html

but don't see anything about either "page" or "size" contained
therein.

There's nothing particularly complex as far as I can tell.  Nothing
more than a straightforward

  import dbm
  with dbm.open("cache", "c") as db:
    for thing in source:
      key = extract_key_as_bytes(thing)
      if key in db:
        data = db[key]
      else:
        data = long_process(thing)
        db[key] = data

The keys can get a bit large (think roughly book-title length), but
not huge. I have 11k records so it seems like it shouldn't be
overwhelming, but this is the second batch where I've had to nuke the
cache and start afresh.  Fortunately I've tooled the code so it can
work incrementally and no more than a hundred or so requests have to
be re-performed.

How does one increas the page-size in a dbm mapping?  Or are there
limits that I should be aware of?

Thanks,

-tkc

PS: FWIW, this is Python 3.6 on FreeBSD in case that exposes any
germane implementation details.









More information about the Python-list mailing list