From donald at stufft.io  Mon Mar 24 22:53:52 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 24 Mar 2014 17:53:52 -0400
Subject: [Import-SIG] [Distutils] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <20140324174807.33712c4d@limelight.wooz.org>
References: <20140324174807.33712c4d@limelight.wooz.org>
Message-ID: <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>


See also https://github.com/pypa/pip/issues/3

Basically prior to PEP420 namespace packages were bad and using them
results in pain sooner or later :( I?m not sure if a good solution yet, perhaps
we can backport PEP420 to PyPI and have namespace packages depend
on that?

On Mar 24, 2014, at 5:48 PM, Barry Warsaw <barry at python.org> wrote:

> Apologies for cross-posting, but this intersects setuptools and the import
> system, and I wanted to be sure it reached the right audience.
> 
> A colleague asked me why a seemingly innocent and common use case for
> developing local versions of system installed packages wasn't working, and I
> was quite perplexed.  As I dug into the problem, more questions than answers
> came up.  I finally (think! I) figured out what is happening, but not so much
> as to why, or what can/should be done about it.
> 
> This person had a local checkout of a package's source, where the package was
> also installed into the system Python.  He wanted to be able to set
> $PYTHONPATH so that the local package wins when he tries to import it.  E.g.:
> 
> % PYTHONPATH=`pwd`/src python3
> 
> but this didn't work because despite the setting of PYTHONPATH, the system
> version of the package was always found first.  The package in question is
> lazr.uri, although other packages with similar layouts will also suffer the
> same problem, which prevents an easy local development of a newer version of
> the package, aside from being a complete head-scratcher.
> 
> The lazr.uri package is intended to be a submodule of the lazr namespace
> package.  As such, the lazr/__init__.py has the old style way of declaring a
> namespace package:
> 
> try:
>    import pkg_resources
>    pkg_resources.declare_namespace(__name__)
> except ImportError:
>    import pkgutil
>    __path__ = pkgutil.extend_path(__path__, __name__)
> 
> and its setup.py declares a namespace package:
> 
> setup(
>    name='lazr.uri',
>    version=__version__,
>    namespace_packages=['lazr'],
>    ...
> 
> One of the things that the Debian "helper" program does when it builds a
> package for the archive is call `$python setup.py install_egg_info`.  It's
> this command that breaks $PYTHONPATH overriding.
> 
> install_egg_info looks at the lazr.uri.egg-info/namespace_packages.txt file,
> in which it finds the string 'lazr', and it proceeds to write a
> lazr-uri-1.0.3-py3.4-nspkg.pth file.  This causes other strange and unexpected
> things to happen:
> 
> % python3
> Python 3.4.0 (default, Mar 22 2014, 22:51:25) 
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.modules['lazr']  
> <module 'lazr'>
>>>> sys.modules['lazr'].__path__  
> ['/usr/lib/python3/dist-packages/lazr']
> 
> It's completely weird that sys.modules would contain a key for 'lazr' when
> that package was never explicitly imported.  Even stranger, because a fake
> module object is stuffed into sys.modules via the .pth file, tracing imports
> with -v gives you no clue as to what's happening.  And while
> sys.modules['lazr'] has an __path__, it has no other attributes.
> 
> I really don't understand what the purpose of the nspkg.pth file is,
> especially for Python 3 namespace packages.
> 
> Here's what the nspkg.pth file contains:
> 
> import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('lazr',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('lazr',types.ModuleType('lazr')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
> 
> The __path__ value is important here because even though you've never
> explicitly imported 'lazr', when you *do* explicitly import 'lazr.uri', the
> existing lazr module object's __path__ takes over, and thus the system
> lazr.uri package is found even though both lazr/ and lazr/uri/ should have
> been found earlier on sys.path (yes, sys.path looks exactly as expected).
> 
> So the presence of the nspkg.pth file breaks $PYTHONPATH overriding.  That
> seems bad. ;)
> 
> If you delete the nspkg.path file, then things work as expected, but even this
> is a little misleading!
> 
> I think the Debian helper is running install_egg_info as a way to determine
> what namespace packages are defined, so that it can actually *remove* the
> parent's __init__.py file and use PEP 420 style namespace packages.  In fact,
> in the Debian python3-lazr.uri binary package, you find no system
> lazr/__init__.py file.  This is why removing the nspkg.pth file works.
> 
> So I thought, why not conditionally define setup(..., namespace_packages) only
> for Python 2?  This doesn't work because the Debian helper will see that no
> namespace packages are defined, and thus it will leave the original
> lazr/__init__.py file in place.  This then breaks $PYTHONPATH overriding too
> because of __path__ extension of the pre-PEP 420 code only *appends* the local
> development path.  IOW, the system import path is the first element of a
> 2-element list on lazr.__path__.  While the local import path is the second
> element, in this case too the local import fails.
> 
> It seems like what you want for Python 3 (and we're talking >= 3.2 here) is
> for there to be neither a nspkg.pth file, nor the lazr/__init__.py file, and
> let PEP 420 do it's thing.  In fact if you set things up this way, $PYTHONPATH
> overriding works exactly as expected.
> 
> Because I don't know why install_egg_info is installing the nspkg.pth file, I
> don't know which component needs to be changed:
> 
> * Change setuptools install_egg_info command to not install an nspkg.pth file
>   even for namespace_package declare packages, at least under Python 3.
>   This behavior seems pretty nasty all by itself because it magically and
>   untraceably installs stripped down module objects in sys.modules when
>   Python first scans the import path.
> 
> * Change the Debian helper to remove the nspkg.pth file, or not call
>   install_egg_info *and* continue to remove <nspkg>/__init__.py in Python 3
>   so as to take advantage of PEP 420.  It's nice to know that PEP 420
>   actually represents something sane. :)
> 
> For added bonus, we have this additional oddity:
> 
> % PYTHONPATH=`pwd`/src python3
> Python 3.4.0 (default, Mar 22 2014, 22:51:25) 
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.modules['lazr']
> <module 'lazr'>
>>>> sys.modules['lazr'].__path__
> ['/usr/lib/python3/dist-packages/lazr']
>>>> import lazr.uri
>>>> lazr.uri.__file__
> '/usr/lib/python3/dist-packages/lazr/uri/__init__.py'
>>>> sys.modules['lazr']
> <module 'lazr' from '/home/barry/projects/ubuntu/lazruri/trusty/src/lazr/__init__.py'>
>>>> sys.modules['lazr'].__path__
> ['/home/barry/projects/ubuntu/lazruri/trusty/src/lazr', '/usr/lib/python3/dist-packages/lazr']
> 
> 
> Notice how importing lazr.uri *replaces* sys.modules['lazr'] with the local
> development one, even though it still imports lazr.uri from the system path.
> I'm not exactly sure how this happens, but I've traced that to
> _LoaderBasics.exec_module()'s call of _call_with_frames_removed(), which
> exec's lazr.uri's code object into that module's __dict__.   Nothing in
> lazr/uri/__init__.py should be doing that, afaict from both visual inspection
> of the code and disassembling the compiled code object.
> 
> Hopefully I've explained the situation correctly and lucidly.  Below I'll
> describe how to set up a reproducible environment on a Debian machine.
> Thoughts and comments are welcome!
> 
> Cheers,
> -Barry
> 
> % sudo apt-get install python3-lazr.uri
> % cd tmp
> % bzr branch lp:lazr.uri trunk
> % cd trunk
> % PYTHONPATH=`pwd`/src python3
> (Then try things at the Python prompt from above.)
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140324/25c7ced8/attachment.sig>

From barry at python.org  Mon Mar 24 23:09:17 2014
From: barry at python.org (Barry Warsaw)
Date: Mon, 24 Mar 2014 18:09:17 -0400
Subject: [Import-SIG] [Distutils] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
Message-ID: <20140324180917.6d42a392@anarchist.wooz.org>

On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote:

>See also https://github.com/pypa/pip/issues/3

Hah, yeah.  I didn't realize --single-version-externally-managed is implied by
--root, and in fact our Debian build scripts often (either explicitly or
implicitly) define --root, as is the case in lazr.uri.

>Basically prior to PEP420 namespace packages were bad and using them results
>in pain sooner or later :( I?m not sure if a good solution yet, perhaps we
>can backport PEP420 to PyPI and have namespace packages depend on that?

Or maybe do a version check before installing this file?  Python 3.3 and newer
will have PEP 420 support, and this file makes no sense in that case.

(Backporting PEP 420 support to Python 3.2 might be useful, though a pain
without importlib.  There's no sense in backporting to Python 3.1, IMHO.)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140324/9c0300c3/attachment.sig>

From bcannon at gmail.com  Tue Mar 25 15:14:40 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Tue, 25 Mar 2014 14:14:40 +0000
Subject: [Import-SIG] [Distutils] nspkg.pth files break $PYTHONPATH
	overrides
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
Message-ID: <CAP1=2W5DoHhL5+1MA7vENxj6u_Y7Y5jq-5UzFRLTmeEbm0+nkQ@mail.gmail.com>

On Mon Mar 24 2014 at 6:09:48 PM, Barry Warsaw <barry at python.org> wrote:

> On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote:
>
> >See also https://github.com/pypa/pip/issues/3
>
> Hah, yeah.  I didn't realize --single-version-externally-managed is
> implied by
> --root, and in fact our Debian build scripts often (either explicitly or
> implicitly) define --root, as is the case in lazr.uri.
>
> >Basically prior to PEP420 namespace packages were bad and using them
> results
> >in pain sooner or later :( I?m not sure if a good solution yet, perhaps we
> >can backport PEP420 to PyPI and have namespace packages depend on that?
>
> Or maybe do a version check before installing this file?  Python 3.3 and
> newer
> will have PEP 420 support, and this file makes no sense in that case.
>
> (Backporting PEP 420 support to Python 3.2 might be useful, though a pain
> without importlib.  There's no sense in backporting to Python 3.1, IMHO.)
>

Importlib is in Python 3.2. You would just need to do the right things to
set up the environment and set __import__ to avoid doubling the stat calls
on a failed import.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140325/1dbd9623/attachment.html>

From pje at telecommunity.com  Tue Mar 25 20:35:57 2014
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 25 Mar 2014 15:35:57 -0400
Subject: [Import-SIG] [Distutils] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <20140324180917.6d42a392@anarchist.wooz.org>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
Message-ID: <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>

On Mon, Mar 24, 2014 at 6:09 PM, Barry Warsaw <barry at python.org> wrote:

> On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote:
>
> >See also https://github.com/pypa/pip/issues/3
>
> Hah, yeah.  I didn't realize --single-version-externally-managed is
> implied by
> --root, and in fact our Debian build scripts often (either explicitly or
> implicitly) define --root, as is the case in lazr.uri.
>
> >Basically prior to PEP420 namespace packages were bad and using them
> results
> >in pain sooner or later :( I'm not sure if a good solution yet, perhaps we
> >can backport PEP420 to PyPI and have namespace packages depend on that?
>
> Or maybe do a version check before installing this file?  Python 3.3 and
> newer
> will have PEP 420 support, and this file makes no sense in that case.
>

That won't *quite* work, because the .pth files are generated for other
types of direct-install distributions.

I think the correct fix would be to change the nspkg.pth magic to check for
PEP 420 support, but unfortunately it seems we may have to use version
checking: on rereading PEP 420 I see there's no 'sys.namespace_packages' or
similar object that can be directly checked for feature support.  :-(
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140325/9f30aed2/attachment.html>

From barry at python.org  Tue Mar 25 20:50:13 2014
From: barry at python.org (Barry Warsaw)
Date: Tue, 25 Mar 2014 15:50:13 -0400
Subject: [Import-SIG] [Distutils] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
 <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
Message-ID: <20140325155013.4f6c1dd9@anarchist.wooz.org>

On Mar 25, 2014, at 03:35 PM, PJ Eby wrote:

>I think the correct fix would be to change the nspkg.pth magic to check for
>PEP 420 support, but unfortunately it seems we may have to use version
>checking: on rereading PEP 420 I see there's no 'sys.namespace_packages' or
>similar object that can be directly checked for feature support.  :-(

There is.  It's *pronounced* sys.namespace_packages, but it's spelled
importlib._bootstrap._NamespaceLoader ;)

http://youtu.be/ehKGlT2EW1Q?t=44s

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140325/282b01a9/attachment.sig>

From pje at telecommunity.com  Tue Mar 25 21:30:56 2014
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 25 Mar 2014 16:30:56 -0400
Subject: [Import-SIG] [Distutils] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <20140325155013.4f6c1dd9@anarchist.wooz.org>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
 <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
 <20140325155013.4f6c1dd9@anarchist.wooz.org>
Message-ID: <CALeMXf7hfUwp73BGmZPsogZRkU-t1xS4sEKMYnjANhLLV0zEhA@mail.gmail.com>

On Tue, Mar 25, 2014 at 3:50 PM, Barry Warsaw <barry at python.org> wrote:

> On Mar 25, 2014, at 03:35 PM, PJ Eby wrote:
>
> >I think the correct fix would be to change the nspkg.pth magic to check
> for
> >PEP 420 support, but unfortunately it seems we may have to use version
> >checking: on rereading PEP 420 I see there's no 'sys.namespace_packages'
> or
> >similar object that can be directly checked for feature support.  :-(
>
> There is.  It's *pronounced* sys.namespace_packages, but it's spelled
> importlib._bootstrap._NamespaceLoader ;)
>

Yeah, well that's not exactly a public attribute, so it's not necessarily a
great way to do it.  But if we did use that, presumably it'd add something
like:

    hnp =
hasattr(sys.modules.get('importlib._bootstrap',None),'_NamespaceLoader');

To the front of the magic, and then prefix all subsequent expression values
with 'not hnp and' in order to prevent them executing if PEP 420 support is
available. (Note: it's checking sys.modules since on any interpreter where
PEP 420 is natively available, the module should *already* be loaded by the
time site.py does its thing.)

And we should probably think about adding sys.namespace_packages or
something of the sort, or at least a proper flag for whether PEP 420
support is available on the platform.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140325/f965c567/attachment.html>

From ericsnowcurrently at gmail.com  Thu Mar 27 04:55:18 2014
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 26 Mar 2014 21:55:18 -0600
Subject: [Import-SIG] [Distutils] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <CALeMXf7hfUwp73BGmZPsogZRkU-t1xS4sEKMYnjANhLLV0zEhA@mail.gmail.com>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
 <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
 <20140325155013.4f6c1dd9@anarchist.wooz.org>
 <CALeMXf7hfUwp73BGmZPsogZRkU-t1xS4sEKMYnjANhLLV0zEhA@mail.gmail.com>
Message-ID: <CALFfu7AWUb3xMQwaB04BKQkzfAOB9ceY6rXjp7HJdEYpNZ8SeA@mail.gmail.com>

On Tue, Mar 25, 2014 at 2:30 PM, PJ Eby <pje at telecommunity.com> wrote:
> On Tue, Mar 25, 2014 at 3:50 PM, Barry Warsaw <barry at python.org> wrote:
>> There is.  It's *pronounced* sys.namespace_packages, but it's spelled
>> importlib._bootstrap._NamespaceLoader ;)
>
>
> Yeah, well that's not exactly a public attribute, so it's not necessarily a
> great way to do it.  But if we did use that, presumably it'd add something
> like:
>
>     hnp =
> hasattr(sys.modules.get('importlib._bootstrap',None),'_NamespaceLoader');

In 3.4 it's called _NamespaceLoader, but in 3.3 it's NamespaceLoader. <ducks>

>
> To the front of the magic, and then prefix all subsequent expression values
> with 'not hnp and' in order to prevent them executing if PEP 420 support is
> available. (Note: it's checking sys.modules since on any interpreter where
> PEP 420 is natively available, the module should *already* be loaded by the
> time site.py does its thing.)
>
> And we should probably think about adding sys.namespace_packages or
> something of the sort, or at least a proper flag for whether PEP 420 support
> is available on the platform.

By "available on the platform" do you mean "Python 3.3+ or has a
backport installed"?  Otherwise you'd just check sys.version*.

-eric

From pje at telecommunity.com  Thu Mar 27 19:35:55 2014
From: pje at telecommunity.com (PJ Eby)
Date: Thu, 27 Mar 2014 14:35:55 -0400
Subject: [Import-SIG] [Distutils] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <CALFfu7AWUb3xMQwaB04BKQkzfAOB9ceY6rXjp7HJdEYpNZ8SeA@mail.gmail.com>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
 <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
 <20140325155013.4f6c1dd9@anarchist.wooz.org>
 <CALeMXf7hfUwp73BGmZPsogZRkU-t1xS4sEKMYnjANhLLV0zEhA@mail.gmail.com>
 <CALFfu7AWUb3xMQwaB04BKQkzfAOB9ceY6rXjp7HJdEYpNZ8SeA@mail.gmail.com>
Message-ID: <CALeMXf6C=uyKukqxc-XMyJdgidA65OYMcKMo5bKiFqe0qNeozg@mail.gmail.com>

On Wed, Mar 26, 2014 at 11:55 PM, Eric Snow <ericsnowcurrently at gmail.com>wrote:

> In 3.4 it's called _NamespaceLoader, but in 3.3 it's NamespaceLoader.
> <ducks>
>

Ouch. That is going to be a really *long* bit of code.  Not like it isn't
already, though.


By "available on the platform" do you mean "Python 3.3+ or has a
> backport installed"?  Otherwise you'd just check sys.version*.
>

I just hate doing version checks when a feature check ought to be
possible.  You never know when it's going to cause trouble.

But yes, certainly being able to handle the backport case would be nice.
When I wrote PEP 402, I was thinking in terms of transitioning the .pth
files to importing a compatibility module.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140327/80eb2dcf/attachment.html>