[pypy-issue] [issue1686] Issue with our zipimporter module

[Brian Wickman]

Brian Wickman <wickman at gmail.com> added the comment:

I monkeypatched _handle_ns in setuptools/pkg_resources to get some debugging info:


python2.7
declare_namespace twitter
_handle_ns(twitter, code.pex/deps) returned None
_handle_ns(twitter, code.pex) returned code.pex/twitter
module[<module 'twitter' from 'code.pex/deps/twitter/__init__.py'>].__path__ is ['code.pex/deps/twitter'] and appending 
code.pex/twitter
calling load_module(twitter)
declare_namespace twitter
module.__path__ is now ['code.pex/deps/twitter', 'code.pex/twitter']
sys.modules[twitter].__path__ is now ['code.pex/deps/twitter', 'code.pex/twitter']
twitter: <module 'twitter' from 'code.pex/twitter/__init__.py'>, twitter.__path__: ['code.pex/deps/twitter', 'code.pex/twitter']
zipimporter: <zipimporter object "code.pex/deps/">, archive: code.pex, prefix: deps/
zipimporter: <zipimporter object "code.pex">, archive: code.pex, prefix: 
---------------------------
pypy
declare_namespace twitter
_handle_ns(twitter, code.pex/deps) returned None
_handle_ns(twitter, code.pex) returned code.pex/twitter
module[<module 'twitter' from 'code.pex/deps/twitter/__init__.py'>].__path__ is ['code.pex/deps/twitter'] and appending 
code.pex/twitter
calling load_module(twitter)
declare_namespace twitter
module.__path__ is now ['code.pex/deps/twitter', 'code.pex/twitter']
sys.modules[twitter].__path__ is now ['code.pex/twitter']
twitter: <module 'twitter' from 'code.pex/twitter/__init__.py'>, twitter.__path__: ['code.pex/twitter']
zipimporter: <zipimporter object at 0x0000000102ce71c8>, archive: code.pex, prefix: deps/
zipimporter: <zipimporter object at 0x0000000102c84098>, archive: code.pex, prefix: 
Traceback (most recent call last):
  File "app_main.py", line 72, in run_toplevel
  File "/Users/wickman/Python/PyPy-2.2.1/lib-python/2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/wickman/Python/PyPy-2.2.1/lib-python/2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "code.pex/__main__.py", line 17, in <module>
    import twitter.aurora
ImportError: No module named twitter.aurora


The code for _handle_ns is here:


def _handle_ns(packageName, path_item):
    """Ensure that named package includes a subpath of path_item (if needed)"""
    importer = get_importer(path_item)
    if importer is None:
        return None
    loader = importer.find_module(packageName)
    if loader is None:
        return None
    module = sys.modules.get(packageName)
    if module is None:
        module = sys.modules[packageName] = imp.new_module(packageName)
        print('setting module(%s) __path__ to []' % module)
        module.__path__ = []
        _set_parent_ns(packageName)
    elif not hasattr(module,'__path__'):
        raise TypeError("Not a package:", packageName)
    handler = _find_adapter(_namespace_handlers, importer)
    subpath = handler(importer,path_item,packageName,module)
    print('_handle_ns(%s, %s) returned %s' % (packageName, path_item, subpath))
    if subpath is not None:
        print('module[%s].__path__ is %s and appending %s' % (module, module.__path__, subpath))
        path = module.__path__
        path.append(subpath)
        print('calling load_module(%s)' % packageName)
        loaded_module = loader.load_module(packageName)
        module.__path__ = path
        print('module.__path__ is now %s' % module.__path__)
        print('sys.modules[%s].__path__ is now %s' % (packageName, sys.modules[packageName].__path__))
    return subpath


It looks like somehow sys.modules[packageName] gets replaced with a new/different version during zipimporter.load_module from the 
one that's getting its __path__ appended to.

________________________________________
PyPy bug tracker <tracker at bugs.pypy.org>
<https://bugs.pypy.org/issue1686>
________________________________________