[New-bugs-announce] [issue33074] dbm corrupts index on macOS (_dbm module)

[Robert Xiao]

New submission from Robert Xiao <nneonneo at gmail.com>:

Environment: Python 3.6.4, macOS 10.12.6

Python 3's dbm appears to corrupt the key index on macOS if objects >4KB are inserted.

Code:

<<<<<<<<<<<
import dbm
import contextlib

with contextlib.closing(dbm.open('test', 'n')) as db:
    for k in range(128):
        db[('%04d' % k).encode()] = b'\0' * (k * 128)

with contextlib.closing(dbm.open('test', 'r')) as db:
    print(len(db))
    print(len(list(db.keys())))
>>>>>>>>>>>

On my machine, I get the following:

<<<<<<<<<<<
94
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    print(len(list(db.keys())))
SystemError: Negative size passed to PyBytes_FromStringAndSize
>>>>>>>>>>>

(The error says PyString_FromStringAndSize on Python 2.x but is otherwise the same). The expected output, which I see on Linux (using gdbm), is

128
128

I get this error with the following Pythons on my system:

/usr/bin/python2.6 - Apple-supplied Python 2.6.9
/usr/bin/python - Apple-supplied Python 2.7.13
/opt/local/bin/python2.7 - MacPorts Python 2.7.14
/usr/local/bin/python - Python.org Python 2.7.13
/usr/local/bin/python3.5 - Python.org Python 3.5.1
/usr/local/bin/python3.6 - Python.org Python 3.6.4

This seems like a very big problem - silent data corruption with no warning. It appears related to issue30388, but in that case they were seeing sporadic failures. The deterministic script above causes failures in every case.

This was discovered after running some code which used shelve (which uses dbm under the hood) in Python 3, but the bug clearly applies to Python 2 as well.

----------
files: test.db
messages: 313809
nosy: nneonneo
priority: normal
severity: normal
status: open
title: dbm corrupts index on macOS (_dbm module)
versions: Python 2.7, Python 3.5, Python 3.6
Added file: https://bugs.python.org/file47484/test.db

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33074>
_______________________________________