[issue33074] dbm corrupts index on macOS (_dbm module)

[Robert Xiao]

Robert Xiao <nneonneo at gmail.com> added the comment:

So we have some other problems then:

(1) It should be documented in dbm, and ideally in shelve, that keys/values over a certain limit might not work. Presently there is no hint that such a limit exists, and until you mentioned it I was unaware that POSIX only required 1023-byte keys and values.
(2) dbm.ndbm should refuse to perform operations that might corrupt the database, or it should be deprecated entirely if this is impossible. A built-in data storage system for Python should not have an easy corruption route, as it is very surprising for users.
(3) It might be worth considering "dbm.sqlite" or somesuch, adapting a SQLite database as a key-value store. The key-value store approach is much simpler than sqlite and appropriate for certain applications, while SQLite itself would provide robustness and correctness. I can volunteer to build such a thing on top of the existing Python SQLite support.
(4) The approach of shelve is incompatible with limited-length values, because shelve's pickles are of an unpredictable length. This suggests that shelve should internally prioritize dumbdbm over ndbm if ndbm cannot guarantee support for arbitrary-length keys/values.
(5) dbm.gnu is not a default, and I can't even work out how to get it enabled with the stock Python installation (i.e. without building my own Python against e.g. Macports gdbm). Is it a problem to ship dbm.gnu as part of the default install, so that it is more feasible to assume its existence? 

Thoughts?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33074>
_______________________________________