[Distutils] bizarre behavior with wheels and namespace packages

[Brian Wickman]

More data:

This works fine with python 3

PYTHONPATH=whl_dist python3 -c 'import twitter.common.python'

but pkg_resource still barfs

PYTHONPATH=whl_dist python3 -c 'import pkg_resources'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1532, in
_find_and_load_unlocked
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2725, in <module>
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 681, in subscribe
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2725, in <lambda>
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2259, in activate
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 1843, in declare_namespace
KeyError: 'twitter'

I did find a related wheel issue:
https://bitbucket.org/pypa/wheel/issue/46/delete-namespace-__init__py-based-on

But this seems to describe the opposite behavior, i.e. that the installer
should be deleting __init__.pys instead of creating them.

In the meantime, my two options are building an installer that populates
__init__.pys, or instead just calling site.addsitedir each time I call
dist.activate().  I'm guessing the latter is preferable as it does not
unnecessarily introduce a pkg_resources dependency.

~brian


On Fri, Mar 7, 2014 at 8:15 PM, Brian Wickman <wickman at gmail.com> wrote:

> Not sure if this belongs on distutils-sig or import-sig, but I'm
> experiencing slightly odd behavior with wheels and namespace packages.
>
> Fetch a namespace package source distribution, e.g. twitter.common.python,
> and build two copies of it, one with bdist_egg and one with bdist_wheel.
>
> This leaves me with:
> twitter.common.python-0.3.1-py2.6.egg
> twitter.common.python-0.3.1-py26-none-any.whl
>
> Explode them into the following directories (respectively):
> egg_dist
> whl_dist
>
> Then attempt:
>
> PYTHONPATH=egg_dist python -c 'import twitter.common.python'
>
> (works)
>
> PYTHONPATH=whl_dist python -c 'import twitter.common.python'
>
> (drumroll)
>
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> ImportError: No module named twitter.common.python
>
> This is slightly unexpected.  Absent from the .whl distribution are the
> __init__.pys with the proper declare_namespace declarations.  Even more
> bizarrely:
>
> PYTHONPATH=whl_dist python -c 'import pkg_resources'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 2736, in <module>
>     add_activation_listener(lambda dist: dist.activate())
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 698, in subscribe
>     callback(dist)
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 2736, in <lambda>
>     add_activation_listener(lambda dist: dist.activate())
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 2274, in activate
>     list(map(declare_namespace,
> self._get_metadata('namespace_packages.txt')))
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 1870, in declare_namespace
>     path = sys.modules[parent].__path__
> KeyError: 'twitter'
>
> So even pkg_resources is surprised as well.  If I do the following:
>
> PYTHONPATH=whl_dist python -c 'import site; site.addsitedir("whl_dist");
> import twitter.common.python'
>
> then it works, since the twitter.common.python-0.3.1-py2.6-nspkg.pth gets
> processed properly and module.__path__s populated accordingly.  I would
> expect pkg_resources to handle this fine but apparently it doesn't.
>
> Any ideas?  Bug in pkg_resources, wheel or PEBKAC?
>
> Thanks!
> ~brian
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140308/ca3d744c/attachment-0001.html>