[New-bugs-announce] [issue19615] "ImportError: dynamic module does not define init function" when deleting and recreating .so files from different machines over NFS

[Edward Catmur]

New submission from Edward Catmur:

foo.c:

#include <Python.h>
static PyMethodDef mth[] = { {NULL, NULL, 0, NULL} };
static struct PyModuleDef mod = { PyModuleDef_HEAD_INIT, "foo", NULL, -1, mth };
PyMODINIT_FUNC PyInit_foo(void) { return PyModule_Create(&mod); }

bar.c:

#include <Python.h>
static PyMethodDef mth[] = { {NULL, NULL, 0, NULL} };
static struct PyModuleDef mod = { PyModuleDef_HEAD_INIT, "bar", NULL, -1, mth };
PyMODINIT_FUNC PyInit_bar(void) { return PyModule_Create(&mod); }

setup.py:

from distutils.core import setup, Extension
setup(name='PackageName',
      ext_modules=[Extension('foo', sources=['foo.c']),
                   Extension('bar', sources=['bar.c'])])

In an NFS mount:

host1$ python setup.py build
host1$ rm *.so; cp build/lib.*/foo*.so .; cp build/lib.*/bar*.so .
host1$ python -c 'import foo; input(); import bar'

While python is waiting for input, on another host in the same directory:

host2$ rm *.so; cp build/lib.*/bar*.so .; cp build/lib.*/foo*.so .

Back on host1:
<Enter>
ImportError: dynamic module does not define init function (PyInit_bar)

Attaching a debugger to Python after the ImportError and calling dlerror() shows the problem:

(gdb) print (char *)dlerror()
$1 = 0xe495210 "/<...>/foo.cpython-34dm.so: undefined symbol: PyInit_bar"

This is because dynload_shlib.c[1] caches dlopen handles by (device and) inode number; but NFS will reuse inode numbers even if a process on a client host has the file open; running lsof on Python, before:

python  16475 ecatmur  mem    REG    0,36    14000 55321147 /<...>/foo.cpython-34dm.so (nfs:/export/user)

and after:

python  16475 ecatmur  mem    REG    0,36          55321147 /<...>/foo.cpython-34dm.so (nfs:/export/user) (path inode=55321161)

Indeed, bar.cpython-34dm.so now has the inode number that Python originally opened foo.cpython-34dm.so under:

host1$ stat -c '%n %i' *.so
bar.cpython-34dm.so 55321147
foo.cpython-34dm.so 55321161

Obviously, this can only happen on a filesystem like NFS where inode numbers can be reused even while a process still has a file open (or mapped).

We encountered this problem in a fairly pathological situation; multiple processes running in two virtualenvs with different copies of a zipped egg (of the same version!) were contending over the ~/.python-eggs directory created by pkg_resources[2] to cache .so files extracted from eggs.  We are working around the situation by setting PYTHON_EGG_CACHE to a virtualenv-specific location, which also fixes the contention issue.  (We should probably work out why the eggs are different, but fixing that is bound into our build/deployment system.)

I'm not sure exactly how to solve or even detect this issue; perhaps looking at the mtime of the .so might work?

If it is decided not to fix the issue it would be useful if _PyImport_GetDynLoadFunc could report the actual dlerror(); this would have saved us quite some time debugging it.  I'll work on a patch to do that.

  1. http://hg.python.org/cpython/file/tip/Python/dynload_shlib.c
  2. https://bitbucket.org/pypa/setuptools/src/ac127a3f46be3037c79f2c4076c7ab221cde21b2/pkg_resources.py?at=default#cl-1040

----------
components: Interpreter Core
messages: 202963
nosy: ecatmur
priority: normal
severity: normal
status: open
title: "ImportError: dynamic module does not define init function" when deleting and recreating .so files from different machines over NFS
type: behavior
versions: Python 3.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue19615>
_______________________________________