From eric at trueblade.com  Mon Apr  2 19:55:54 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Mon, 02 Apr 2012 13:55:54 -0400
Subject: [Import-SIG] Replacement PEP for namespace packages
Message-ID: <4F79E82A.7090508@trueblade.com>

Just in case anyone thinks I've fallen off the face of the earth: I
haven't. I got sick after PyCon (and if I find out which one of you got
me sick ...) and I'm behind on it. I hope to get a good draft of it
ready this week.

Eric.

From yselivanov.ml at gmail.com  Tue Apr 10 01:22:36 2012
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Mon, 9 Apr 2012 19:22:36 -0400
Subject: [Import-SIG] Replacement PEP for namespace packages
In-Reply-To: <4F79E82A.7090508@trueblade.com>
References: <4F79E82A.7090508@trueblade.com>
Message-ID: <FBBF837D-744A-467F-9901-FCD00F63C7D3@gmail.com>

On 2012-04-02, at 1:55 PM, Eric V. Smith wrote:

> Just in case anyone thinks I've fallen off the face of the earth: I
> haven't. I got sick after PyCon (and if I find out which one of you got
> me sick ...) and I'm behind on it. I hope to get a good draft of it
> ready this week.

Hello Eric,

How is it going with the PEP?

(The reason I'm asking is because we want to backport the accepted 
approach to 3.2 and start using it...)

-
Yury

From eric at trueblade.com  Thu Apr 12 14:04:53 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 12 Apr 2012 08:04:53 -0400
Subject: [Import-SIG] Replacement PEP for namespace packages
In-Reply-To: <FBBF837D-744A-467F-9901-FCD00F63C7D3@gmail.com>
References: <4F79E82A.7090508@trueblade.com>
	<FBBF837D-744A-467F-9901-FCD00F63C7D3@gmail.com>
Message-ID: <4F86C4E5.5080102@trueblade.com>

On 04/09/2012 07:22 PM, Yury Selivanov wrote:

> How is it going with the PEP?

I'm reviewing the email threads from the last round of discussions,
making sure I have everything covered. I hope to have a draft early next
week.

> (The reason I'm asking is because we want to backport the accepted 
> approach to 3.2 and start using it...)

Our plan is to wait until Brett merges importlib as the default import
mechanism, so that we can work entirely (or at least mostly) in Python.
Any port to 3.2 is going to be an entirely separate effort, all in C.
But I support it!

Eric.


From brett at python.org  Thu Apr 12 17:34:32 2012
From: brett at python.org (Brett Cannon)
Date: Thu, 12 Apr 2012 11:34:32 -0400
Subject: [Import-SIG] Replacement PEP for namespace packages
In-Reply-To: <4F86C4E5.5080102@trueblade.com>
References: <4F79E82A.7090508@trueblade.com>
	<FBBF837D-744A-467F-9901-FCD00F63C7D3@gmail.com>
	<4F86C4E5.5080102@trueblade.com>
Message-ID: <CAP1=2W7xYrGvyyosVzkx3sJZ-W2_OzL6hYf_V=pOO5Z_RHj-uA@mail.gmail.com>

On Thu, Apr 12, 2012 at 08:04, Eric V. Smith <eric at trueblade.com> wrote:

> On 04/09/2012 07:22 PM, Yury Selivanov wrote:
>
> > How is it going with the PEP?
>
> I'm reviewing the email threads from the last round of discussions,
> making sure I have everything covered. I hope to have a draft early next
> week.
>
> > (The reason I'm asking is because we want to backport the accepted
> > approach to 3.2 and start using it...)
>
> Our plan is to wait until Brett merges importlib as the default import
> mechanism, so that we can work entirely (or at least mostly) in Python.
>

That should be happening soon. Keep an eye on python-dev for an email about
it once the patch lands (have one more thing to work out in default and
then use in my branch before it can land).


> Any port to 3.2 is going to be an entirely separate effort, all in C.
> But I support it!
>

I don't think it necessarily has to be that way. If you either use your own
importer or override builtins.__import__ with importlib.__import__ and
tweak some things under the hood you can actually match behaviour at a
roughly 5% import performance cost compared to what 3.3 will have for
import.

-Brett


>
> Eric.
>
> _______________________________________________
> Import-SIG mailing list
> Import-SIG at python.org
> http://mail.python.org/mailman/listinfo/import-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120412/eba32864/attachment.html>

From eric at trueblade.com  Thu Apr 19 22:18:21 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 19 Apr 2012 16:18:21 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
Message-ID: <4F90730D.1040808@trueblade.com>

This reflects (I hope!) the discussions at PyCon. My plan is to produce
an implementation based on the importlib code, and then flush out pieces
of the PEP.

In particular, I want to make sure the PEP addresses the various
objections that were raised, especially by Nick.

Eric.


From brett at python.org  Thu Apr 19 23:08:43 2012
From: brett at python.org (Brett Cannon)
Date: Thu, 19 Apr 2012 17:08:43 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F90730D.1040808@trueblade.com>
References: <4F90730D.1040808@trueblade.com>
Message-ID: <CAP1=2W4rOR9_FuyHT6=9VscsOn5Lx6w4Tpyq_nRNseGvh0o1yg@mail.gmail.com>

On Thu, Apr 19, 2012 at 16:18, Eric V. Smith <eric at trueblade.com> wrote:

> This reflects (I hope!) the discussions at PyCon. My plan is to produce
> an implementation based on the importlib code, and then flush out pieces
> of the PEP.
>
> In particular, I want to make sure the PEP addresses the various
> objections that were raised, especially by Nick.
>

Obviously thanks for writing this up, Eric! I have the following comments
(some of which I would fix myself but I lack hg repo access ATM)  ...

In Terminology, can you put the terms when you define them in quotes, e.g. 'The
term "distribution" refers to ...'?

"setuptools provides a similar function pkg_resources.declare_namespace"
should either have a "named" added in there or a comma.

"As vendors might chose(sic) to".

You should mention that this will do away with the ImportWarning of
discovering a directory lacking an __init__.py file.

As for the effects on path hooks, there are none. =) It's actually the
finders that they return which need to change. Either finders need to be
updated to return something other than None to signal they have a directory
which works for the name (maybe the string for what should go into
__path__?) or another method on finders which is called if
finder.find_module() returns None (like finder.find_namespace() which
returns the directory name or None). Then you need to update
importlib._bootstrap.PathFinder to handle one of the two approaches to
create the module and set it with some __loader__ (which really doesn't
need to do much more than construct a module with the proper attributes
since there is nothing to execute) like
importlib.machinery.NamespaceLoader(name, *paths). Using a specific class
in import already has precedence thanks to NullImporter.

If you want performance then you go with the returning of a string by
finder.find_module() since the finder can keep track of finding a directory
w/o an __init__.py when it tries looking for a module. Import can do a
hasattr check on non-None return values to decide if it got back a loader
or a path for a namespace. If you don't like what the return value to mean
based on it being None or having a specific attribute then you would want
the new method at the (potential) cost of another stat call. Or maybe
someone can think of some other approach.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120419/bf2a4baa/attachment.html>

From ericsnowcurrently at gmail.com  Thu Apr 19 23:21:32 2012
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 19 Apr 2012 15:21:32 -0600
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F90730D.1040808@trueblade.com>
References: <4F90730D.1040808@trueblade.com>
Message-ID: <CALFfu7CZNo2kX2xmqVzv3R4oG6rg3xvPSY8MD7QhBogsOketiw@mail.gmail.com>

On Thu, Apr 19, 2012 at 2:18 PM, Eric V. Smith <eric at trueblade.com> wrote:
> This reflects (I hope!) the discussions at PyCon. My plan is to produce
> an implementation based on the importlib code, and then flush out pieces
> of the PEP.
>
> In particular, I want to make sure the PEP addresses the various
> objections that were raised, especially by Nick.

Nice work, Eric.  PEP 420 is quite clear.  I appreciate that not many
words are spent on contrasting it with PEP 402.  I agree that the PEP
needs to be clear on Nick's concerns, one way or the other (especially
as they relate to PEP 395).  I don't recall any satisfactory
resolution on that.  Looking forward to hearing more on this.

-eric

p.s.  how often do the PEPs get rebuilt?  I saw the PEP as it came
across the commits list, but it's not showing up on the site.

From eric at trueblade.com  Fri Apr 20 00:10:40 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 19 Apr 2012 18:10:40 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CAP1=2W4rOR9_FuyHT6=9VscsOn5Lx6w4Tpyq_nRNseGvh0o1yg@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CAP1=2W4rOR9_FuyHT6=9VscsOn5Lx6w4Tpyq_nRNseGvh0o1yg@mail.gmail.com>
Message-ID: <4F908D60.9090109@trueblade.com>

On 4/19/2012 5:08 PM, Brett Cannon wrote:
> In Terminology, can you put the terms when you define them in quotes,
> e.g. 'The term "distribution" refers to ...'?
> 
> "setuptools provides a similar function pkg_resources.declare_namespace"
> should either have a "named" added in there or a comma.
> 
> "As vendors might chose(sic) to".

I've made these grammar changes. I'll update based on the rest of your
comments tomorrow.

Thanks!

Eric.

> You should mention that this will do away with the ImportWarning of
> discovering a directory lacking an __init__.py file.
> 
> As for the effects on path hooks, there are none. =) It's actually the
> finders that they return which need to change. Either finders need to be
> updated to return something other than None to signal they have a
> directory which works for the name (maybe the string for what should go
> into __path__?) or another method on finders which is called if
> finder.find_module() returns None (like finder.find_namespace() which
> returns the directory name or None). Then you need to update
> importlib._bootstrap.PathFinder to handle one of the two approaches to
> create the module and set it with some __loader__ (which really doesn't
> need to do much more than construct a module with the proper attributes
> since there is nothing to execute) like
> importlib.machinery.NamespaceLoader(name, *paths). Using a specific
> class in import already has precedence thanks to NullImporter.
> 
> If you want performance then you go with the returning of a string by
> finder.find_module() since the finder can keep track of finding a
> directory w/o an __init__.py when it tries looking for a module. Import
> can do a hasattr check on non-None return values to decide if it got
> back a loader or a path for a namespace. If you don't like what the
> return value to mean based on it being None or having a specific
> attribute then you would want the new method at the (potential) cost of
> another stat call. Or maybe someone can think of some other approach.


From eric at trueblade.com  Fri Apr 20 00:59:56 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 19 Apr 2012 18:59:56 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CAP1=2W4rOR9_FuyHT6=9VscsOn5Lx6w4Tpyq_nRNseGvh0o1yg@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CAP1=2W4rOR9_FuyHT6=9VscsOn5Lx6w4Tpyq_nRNseGvh0o1yg@mail.gmail.com>
Message-ID: <4F9098EC.4010007@trueblade.com>

On 4/19/2012 5:08 PM, Brett Cannon wrote:

> You should mention that this will do away with the ImportWarning of
> discovering a directory lacking an __init__.py file.

Done.

> As for the effects on path hooks, there are none. =) It's actually the
> finders that they return which need to change. Either finders need to be
> updated to return something other than None to signal they have a
> directory which works for the name (maybe the string for what should go
> into __path__?) or another method on finders which is called if
> finder.find_module() returns None (like finder.find_namespace() which
> returns the directory name or None). Then you need to update
> importlib._bootstrap.PathFinder to handle one of the two approaches to
> create the module and set it with some __loader__ (which really doesn't
> need to do much more than construct a module with the proper attributes
> since there is nothing to execute) like
> importlib.machinery.NamespaceLoader(name, *paths). Using a specific
> class in import already has precedence thanks to NullImporter.
> 
> If you want performance then you go with the returning of a string by
> finder.find_module() since the finder can keep track of finding a
> directory w/o an __init__.py when it tries looking for a module. Import
> can do a hasattr check on non-None return values to decide if it got
> back a loader or a path for a namespace. If you don't like what the
> return value to mean based on it being None or having a specific
> attribute then you would want the new method at the (potential) cost of
> another stat call. Or maybe someone can think of some other approach.

Changing finder.find_module() to return a string seems the best thing to do.

Barry and I (and hopefully Jason Coombs) are going to try and get
together and sprint on this in the near future. I might wait to update
the PEP on the affect on finders until we're done.

Thanks again.

Eric.

From ncoghlan at gmail.com  Fri Apr 20 05:56:50 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Apr 2012 13:56:50 +1000
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F90730D.1040808@trueblade.com>
References: <4F90730D.1040808@trueblade.com>
Message-ID: <CADiSq7c_c-NOM3CC4DsY77WSV=HB5K8k=Azj-m-aN=-vmBdXPw@mail.gmail.com>

On Fri, Apr 20, 2012 at 6:18 AM, Eric V. Smith <eric at trueblade.com> wrote:
> This reflects (I hope!) the discussions at PyCon. My plan is to produce
> an implementation based on the importlib code, and then flush out pieces
> of the PEP.

This paragraph in the "Rationale" section is confusing:

"Namespace packages need to be installed in one of two ways: either
all portions of a namespace will be combined into a single directory
(and therefore a single entry in sys.path), or each portion will be
installed in its own directory (and each portion will have a distinct
sys.path entry)."

I would combine this with the following paragraph to make a single
cohesive explanation of the problem that needs to be solved:

"Namespace packages are designed to support being split across
multiple directories (and hence found via multiple sys.path entries).
In this configuration, it doesn't matter if multiple portions all
provide an __init__.py file, so long as each portion correctly
initialises the namespace package. However, Linux distribution vendors
(amongst others) prefer to combine the separate portions and install
them all into the *same* filesystem directory. This creates a
potential for conflict, as the portions are now attempting to provide
the *same* file on the target system - something that is not allowed
by many package managers. Allowing implicit namespace packages means
that the requirement to provide an __init__.py file can be dropped
completely, and affected portions can be installed into a common
directory or split across multiple directories as distributions see
fit."

> In particular, I want to make sure the PEP addresses the various
> objections that were raised, especially by Nick.

Yep. I'm happy with the conclusions we reached in the previous
discussion, but PEP 420 does need to describe them. Here's the gist of
it for the four points listed:

- for the first point, "practicality beats purity" pretty much carries
the day as far the Zen goes

- for the second point, the minor backwards compatibility risks are
acknowledged and accepted. My initial objection was based on a
misunderstanding of the consensus proposal. Once it was clarified that
the only "incompatibility" is that an import may now succeed where it
previously would have failed, I was no longer concerned. In contrast
to PEP 402, PEP 420 deliberately chooses to preserve consistent
behaviour of "import foo; import foo.bar" and "import foo.bar; import
foo", seeing that as being more important than preventing the
successful import of an empty (or otherwise non-package) subdirectory
of a sys.path location. This does mean some try/except import blocks
may need to updated to check the imported module or package for an
expected attribute or subpackage rather than just checking that the
import works, but has the major advantage of making the revised import
model much cleaner and easier to understand.

- the final two points will be addressed by having PEP 395 propose the
production of better *error messages* rather than introducing any
additional magic to the initialisation of sys.path[0] (see
http://mail.python.org/pipermail/import-sig/2012-March/000442.html).
The "are we in a package subdirectory?" heuristic mentioned in that
message will be based on this suggestion from Eric Snow:
http://mail.python.org/pipermail/import-sig/2012-March/000438.html

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia

From ncoghlan at gmail.com  Fri Apr 20 06:04:59 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Apr 2012 14:04:59 +1000
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F90730D.1040808@trueblade.com>
References: <4F90730D.1040808@trueblade.com>
Message-ID: <CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>

On Fri, Apr 20, 2012 at 6:18 AM, Eric V. Smith <eric at trueblade.com> wrote:
> This reflects (I hope!) the discussions at PyCon. My plan is to produce
> an implementation based on the importlib code, and then flush out pieces
> of the PEP.
>
> In particular, I want to make sure the PEP addresses the various
> objections that were raised, especially by Nick.

One other thing I noticed: "There is no mechanism to recompute the
__path__ once a namespace package has been created."

This isn't really true - pkgutil.extend_path() can still be used to
update a namespace package path. Perhaps change it to:

"There is no mechanism to automatically recompute the __path__ if
sys.path is altered after a namespace package has already been
created. However, existing namespace utilities (like
pkgutil.extend_path()) can be used to update them explicitly if
desired."

Also, as a general matter of readability, adding double backticks
around attributes, functions and filenames to get them displayed in
monospace can be quite helpful.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia

From eric at trueblade.com  Fri Apr 20 12:15:30 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 20 Apr 2012 06:15:30 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>
Message-ID: <4F913742.2070302@trueblade.com>

On 4/20/2012 12:04 AM, Nick Coghlan wrote:
> On Fri, Apr 20, 2012 at 6:18 AM, Eric V. Smith <eric at trueblade.com> wrote:
>> This reflects (I hope!) the discussions at PyCon. My plan is to produce
>> an implementation based on the importlib code, and then flush out pieces
>> of the PEP.
>>
>> In particular, I want to make sure the PEP addresses the various
>> objections that were raised, especially by Nick.
> 
> One other thing I noticed: "There is no mechanism to recompute the
> __path__ once a namespace package has been created."
> 
> This isn't really true - pkgutil.extend_path() can still be used to
> update a namespace package path. Perhaps change it to:
> 
> "There is no mechanism to automatically recompute the __path__ if
> sys.path is altered after a namespace package has already been
> created. However, existing namespace utilities (like
> pkgutil.extend_path()) can be used to update them explicitly if
> desired."

Done. Thanks!

> Also, as a general matter of readability, adding double backticks
> around attributes, functions and filenames to get them displayed in
> monospace can be quite helpful.

Agreed. That's a work in progress.

Eric.


From eric at trueblade.com  Fri Apr 20 12:21:31 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 20 Apr 2012 06:21:31 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CADiSq7c_c-NOM3CC4DsY77WSV=HB5K8k=Azj-m-aN=-vmBdXPw@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CADiSq7c_c-NOM3CC4DsY77WSV=HB5K8k=Azj-m-aN=-vmBdXPw@mail.gmail.com>
Message-ID: <4F9138AB.40802@trueblade.com>

On 4/19/2012 11:56 PM, Nick Coghlan wrote:
> On Fri, Apr 20, 2012 at 6:18 AM, Eric V. Smith <eric at trueblade.com> wrote:
>> This reflects (I hope!) the discussions at PyCon. My plan is to produce
>> an implementation based on the importlib code, and then flush out pieces
>> of the PEP.
> 
> This paragraph in the "Rationale" section is confusing:
> 
> "Namespace packages need to be installed in one of two ways: either
> all portions of a namespace will be combined into a single directory
> (and therefore a single entry in sys.path), or each portion will be
> installed in its own directory (and each portion will have a distinct
> sys.path entry)."
> 
> I would combine this with the following paragraph to make a single
> cohesive explanation of the problem that needs to be solved:
> 
> "Namespace packages are designed to support being split across
> multiple directories (and hence found via multiple sys.path entries).
> In this configuration, it doesn't matter if multiple portions all
> provide an __init__.py file, so long as each portion correctly
> initialises the namespace package. However, Linux distribution vendors
> (amongst others) prefer to combine the separate portions and install
> them all into the *same* filesystem directory. This creates a
> potential for conflict, as the portions are now attempting to provide
> the *same* file on the target system - something that is not allowed
> by many package managers. Allowing implicit namespace packages means
> that the requirement to provide an __init__.py file can be dropped
> completely, and affected portions can be installed into a common
> directory or split across multiple directories as distributions see
> fit."

That does read much better. Thanks.

>> In particular, I want to make sure the PEP addresses the various
>> objections that were raised, especially by Nick.
> 
> Yep. I'm happy with the conclusions we reached in the previous
> discussion, but PEP 420 does need to describe them. Here's the gist of
> it for the four points listed:

<discussion deleted>

I'll add these after I go back and re-read the original thread.

Eric.


From pje at telecommunity.com  Sat Apr 21 19:06:23 2012
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 21 Apr 2012 13:06:23 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>
Message-ID: <CALeMXf4Omdhh5iPApM23UTsrcNm3CR_eJzKes6ta+XBerKz5mQ@mail.gmail.com>

On Fri, Apr 20, 2012 at 12:04 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> "There is no mechanism to automatically recompute the __path__ if
> sys.path is altered after a namespace package has already been
> created. However, existing namespace utilities (like
> pkgutil.extend_path()) can be used to update them explicitly if
> desired."
>

Btw, was there ever an explicit rejection of the "namespace package
__path__ is an auto-updating iterable instead of a list" approach, or did
it even come up in the consensus discussion?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120421/ef68c2e9/attachment.html>

From ericsnowcurrently at gmail.com  Sat Apr 21 20:49:16 2012
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 21 Apr 2012 12:49:16 -0600
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CALeMXf4Omdhh5iPApM23UTsrcNm3CR_eJzKes6ta+XBerKz5mQ@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>
	<CALeMXf4Omdhh5iPApM23UTsrcNm3CR_eJzKes6ta+XBerKz5mQ@mail.gmail.com>
Message-ID: <CALFfu7CpPWEanuyD8WfobQCitB_qz=FX08XjiL_0L9WT+1X28w@mail.gmail.com>

On Sat, Apr 21, 2012 at 11:06 AM, PJ Eby <pje at telecommunity.com> wrote:
> Btw, was there ever an explicit rejection of the "namespace package __path__
> is an auto-updating iterable instead of a list" approach, or did it even
> come up in the consensus discussion?

Pretty sure it didn't come up, but it sounds like Eric Smith has
considered it.  PEP 420 currently has this to say:

  "There is no mechanism to automatically recompute the __path__ if
sys.path is altered after a namespace package has already been
created. However, existing namespace utilities (like
pkgutil.extend_path) can be used to update them explicitly if
desired." [1]

-eric

[1] http://www.python.org/dev/peps/pep-0420/#id9

From martin at v.loewis.de  Sat Apr 21 22:50:23 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 21 Apr 2012 22:50:23 +0200
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CALFfu7CpPWEanuyD8WfobQCitB_qz=FX08XjiL_0L9WT+1X28w@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>	<CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>	<CALeMXf4Omdhh5iPApM23UTsrcNm3CR_eJzKes6ta+XBerKz5mQ@mail.gmail.com>
	<CALFfu7CpPWEanuyD8WfobQCitB_qz=FX08XjiL_0L9WT+1X28w@mail.gmail.com>
Message-ID: <4F931D8F.2030305@v.loewis.de>

Am 21.04.2012 20:49, schrieb Eric Snow:
> On Sat, Apr 21, 2012 at 11:06 AM, PJ Eby <pje at telecommunity.com> wrote:
>> Btw, was there ever an explicit rejection of the "namespace package __path__
>> is an auto-updating iterable instead of a list" approach, or did it even
>> come up in the consensus discussion?
> 
> Pretty sure it didn't come up, but it sounds like Eric Smith has
> considered it. 

There was a sort of bulk-rejection of "fancy features", IIRC. It wasn't
clear to us which of the many additional features of PEP 402 was really
important to you, so the consensus was to start with the minimum, and
extend as actual use cases become apparent.

For some of the PEP 402 features, we identified "concurrent versions"
as the use case (i.e. pkg_resources.require). The consensus was that
this use case can be ignored.

Eric is right that the specific question of a dynamic __path__ was not
discussed.

Regards,
Martin

From eric at trueblade.com  Sun Apr 22 03:06:45 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Sat, 21 Apr 2012 21:06:45 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F931D8F.2030305@v.loewis.de>
References: <4F90730D.1040808@trueblade.com>	<CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>	<CALeMXf4Omdhh5iPApM23UTsrcNm3CR_eJzKes6ta+XBerKz5mQ@mail.gmail.com>
	<CALFfu7CpPWEanuyD8WfobQCitB_qz=FX08XjiL_0L9WT+1X28w@mail.gmail.com>
	<4F931D8F.2030305@v.loewis.de>
Message-ID: <4F9359A5.2060800@trueblade.com>

On 4/21/2012 4:50 PM, "Martin v. L?wis" wrote:
> Am 21.04.2012 20:49, schrieb Eric Snow:
>> On Sat, Apr 21, 2012 at 11:06 AM, PJ Eby <pje at telecommunity.com> wrote:
>>> Btw, was there ever an explicit rejection of the "namespace package __path__
>>> is an auto-updating iterable instead of a list" approach, or did it even
>>> come up in the consensus discussion?

What's the use case for this?

>> Pretty sure it didn't come up, but it sounds like Eric Smith has
>> considered it. 
> 
> There was a sort of bulk-rejection of "fancy features", IIRC. It wasn't
> clear to us which of the many additional features of PEP 402 was really
> important to you, so the consensus was to start with the minimum, and
> extend as actual use cases become apparent.

I don't recall this issue specifically, but I agree with Martin that
we're trying to start with a minimal feature set.

> Eric is right that the specific question of a dynamic __path__ was not
> discussed.

Furthermore, given how __path__ is built, by one-at-a-time remembering
the path entries that have a foo directory but no foo/__init__.py, I'm
not sure how you'd turn that into some auto-updating iterable.

Eric.


From ncoghlan at gmail.com  Sun Apr 22 07:26:06 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 22 Apr 2012 15:26:06 +1000
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F9359A5.2060800@trueblade.com>
References: <4F90730D.1040808@trueblade.com>
	<CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>
	<CALeMXf4Omdhh5iPApM23UTsrcNm3CR_eJzKes6ta+XBerKz5mQ@mail.gmail.com>
	<CALFfu7CpPWEanuyD8WfobQCitB_qz=FX08XjiL_0L9WT+1X28w@mail.gmail.com>
	<4F931D8F.2030305@v.loewis.de> <4F9359A5.2060800@trueblade.com>
Message-ID: <CADiSq7dTOu4S0cVwbtZcfuWhuO7stcShLbZa+BE0ZKy3LLhfmQ@mail.gmail.com>

On Sun, Apr 22, 2012 at 11:06 AM, Eric V. Smith <eric at trueblade.com> wrote:
> Furthermore, given how __path__ is built, by one-at-a-time remembering
> the path entries that have a foo directory but no foo/__init__.py, I'm
> not sure how you'd turn that into some auto-updating iterable.

You just have to remember all your namespace packages somewhere and
then use a list subclass that triggers a rescan whenever the contents
change.

Personally, I'm happier with the basic behaviour being that
dynamically updating sys.path while the program is running can be a
bit hit-or-miss in terms of what recognises the change.

Longer term, rather than introducing magical side effects for sys.path
manipulation, I think the better solution is to expose a more
object-oriented API for manipulating the import system state that
takes care of maintaining the state invariants, invalidating caches
when appropriate and triggering updates to package __path__ entries.
Hence, PEP 406 (currently deferred) and its import engine API.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia

From pje at telecommunity.com  Sun Apr 22 23:10:27 2012
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 22 Apr 2012 17:10:27 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CADiSq7dTOu4S0cVwbtZcfuWhuO7stcShLbZa+BE0ZKy3LLhfmQ@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CADiSq7eO0xS_u3XArW_x601fkmxz+7izfkSqwsxNODaOQ1fLoQ@mail.gmail.com>
	<CALeMXf4Omdhh5iPApM23UTsrcNm3CR_eJzKes6ta+XBerKz5mQ@mail.gmail.com>
	<CALFfu7CpPWEanuyD8WfobQCitB_qz=FX08XjiL_0L9WT+1X28w@mail.gmail.com>
	<4F931D8F.2030305@v.loewis.de> <4F9359A5.2060800@trueblade.com>
	<CADiSq7dTOu4S0cVwbtZcfuWhuO7stcShLbZa+BE0ZKy3LLhfmQ@mail.gmail.com>
Message-ID: <CALeMXf4cXKvL8zX7iz823itkj-0NcozA7e+YGUsP8poWf=+y2Q@mail.gmail.com>

On Sun, Apr 22, 2012 at 1:26 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Sun, Apr 22, 2012 at 11:06 AM, Eric V. Smith <eric at trueblade.com>
> wrote:
> > Furthermore, given how __path__ is built, by one-at-a-time remembering
> > the path entries that have a foo directory but no foo/__init__.py, I'm
> > not sure how you'd turn that into some auto-updating iterable.
>
> You just have to remember all your namespace packages somewhere and
> then use a list subclass that triggers a rescan whenever the contents
> change.
>

Not necessary if you set __path__ to an iterable that caches a tuple of its
parent package __path__ (or sys.path), and compares that against the
current value before iterating.  If it's changed, you walk the parent and
rescan, otherwise iterate over your cached value.  I posted a sketch to
Python-Dev the first time 402 discussion happened there.

The consequences of making namespace package __path__ iterable are less
problematic, I believe, than changing the type of sys.path: almost no code
manipulates __path__ as anything but an iterable, and code that does is
broken for namespace packages anyway, because accessing specific offsets
won't give you what you think you're looking for.  So you get noisy
breakage instead of quiet breakage in such cases (as would happen with
using lists for __path__).

If for some reason you want to explicitly change a namespace package's
__path__, you could just reset __path__ to list(__path__), and proceed from
there -- which is the recommended idiom for using extend_path, anyway.


Personally, I'm happier with the basic behaviour being that
> dynamically updating sys.path while the program is running can be a
> bit hit-or-miss in terms of what recognises the change.
>

pkg_resources supports dynamic updating today, so the idea here was to make
it possible to do away with that.  (It only supports updating if it's the
one doing the sys.path manipulation, however.)

I think there should be *some* blessed API(s) to force the updating,
though, even if it's not automatic or dynamic.  extend_path() really isn't
the right tool for the job.

The main argument in favor of automatic updating is that it more closely
matches naive expectations of users coming from other languages.  (Although
to be honest I'm not 100% certain that those other languages actually do
change their lookups that dynamically.)

Anyway, the sketch (using PEP 402's importer protocol; not updated for 420)
was something like:

class VirtualPath:
    __slots__ = ('__name__', '_parent', '_last_seen', '_path')

    def __init__(self, name, parent_path):
        self.__name__ = name
        self._parent = parent_path
        self._path = self._last_seen = ()

    def _fail(self, *args, **kw):
        raise TypeError(self.__name__+" is a virtual package")

    __getitem__ = __setitem__ = __delitem__ = append = extend = insert =
_fail

    def _calculate(self):
        with _ImportLockContext():
            parent = tuple(self._parent)
            if parent != self._last_seen:
                items = []
                name = self.__name__
                for entry in parent:
                    importer = get_importer(entry)
                    if hasattr(importer, 'get_subpath'):
                        item = importer.get_subpath(name)
                        if item is not None:
                            items.append(item)
                self._last_seen = parent
                self._path = tuple(items)
            return self._path

    def __iter__(self):
        return iter(self._calculate())

    def __len__(self):
        return len(self._calculate())

    def __repr__(self):
        return "VirtualPath" + repr((self.__name__, self._parent))

    def __contains__(self, item):
        return item in self._calculate()


Using these objects in place of lists for __path__ objects would then do
the trick.

(And of course, you'd want to change "Virtual" to "Namespace" throughout, I
suppose.  ;-) )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120422/d5f4ae8a/attachment.html>

From fuzzyman at gmail.com  Mon Apr 23 01:51:59 2012
From: fuzzyman at gmail.com (Michael Foord)
Date: Mon, 23 Apr 2012 00:51:59 +0100
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F90730D.1040808@trueblade.com>
References: <4F90730D.1040808@trueblade.com>
Message-ID: <CAKCKLWxO2v44Q8j+yG80aywtMtYC7BXFU9DHvnt1NV0-RPtmvQ@mail.gmail.com>

On 19 April 2012 21:18, Eric V. Smith <eric at trueblade.com> wrote:

> This reflects (I hope!) the discussions at PyCon. My plan is to produce
> an implementation based on the importlib code, and then flush out pieces
> of the PEP.
>
> In particular, I want to make sure the PEP addresses the various
> objections that were raised, especially by Nick.
>
>
So a namespace package is a directory (tree) on sys.path. For a standard
Python install how will these be installed?

If you need to install "foo.bar" and "foo.baz" will distutils and packaging
do the right thing? (And what specifically is the right thing for Python's
own package management tools - merging the namespace packages or keeping
them separate somehow?)

setuptools creates a new directory for each installed package and adds this
directory to sys.path using pth files. It's a bit of a hack, but it allows
namespace packages to co-exist.

Michael



> Eric.
>
> _______________________________________________
> Import-SIG mailing list
> Import-SIG at python.org
> http://mail.python.org/mailman/listinfo/import-sig
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120423/20279c6d/attachment.html>

From pje at telecommunity.com  Mon Apr 23 02:29:58 2012
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 22 Apr 2012 20:29:58 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CAKCKLWxO2v44Q8j+yG80aywtMtYC7BXFU9DHvnt1NV0-RPtmvQ@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CAKCKLWxO2v44Q8j+yG80aywtMtYC7BXFU9DHvnt1NV0-RPtmvQ@mail.gmail.com>
Message-ID: <CALeMXf61w4Vn9P+LgGXGg+r8L9xfXDcLHCSkHFVMEw=m983rDw@mail.gmail.com>

On Sun, Apr 22, 2012 at 7:51 PM, Michael Foord <fuzzyman at gmail.com> wrote:

>
>
> On 19 April 2012 21:18, Eric V. Smith <eric at trueblade.com> wrote:
>
>> This reflects (I hope!) the discussions at PyCon. My plan is to produce
>> an implementation based on the importlib code, and then flush out pieces
>> of the PEP.
>>
>> In particular, I want to make sure the PEP addresses the various
>> objections that were raised, especially by Nick.
>>
>>
> So a namespace package is a directory (tree) on sys.path. For a standard
> Python install how will these be installed?
>
> If you need to install "foo.bar" and "foo.baz" will distutils and
> packaging do the right thing? (And what specifically is the right thing for
> Python's own package management tools - merging the namespace packages or
> keeping them separate somehow?)
>

I don't know about 3.x distutils or packaging specifically, but I do know
that 2.x distutils will install packages compatibly with this approach if
you list the child packages but NOT the namespace package in your setup.py.
 So if one distribution lists 'foo.bar' and the other lists 'foo.baz', but
*neither* lists 'foo', then the subpackages will be installed without a
foo/__init__.py, and that will make it work.

If packaging and 3.x distutils inherit this behavior from the 2.x
distutils, then that would be the simplest way to do it.  (And if you
install to different directories, the parts will get merged.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120422/22c5467e/attachment-0001.html>

From ncoghlan at gmail.com  Mon Apr 23 03:08:55 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Apr 2012 11:08:55 +1000
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <CAKCKLWxO2v44Q8j+yG80aywtMtYC7BXFU9DHvnt1NV0-RPtmvQ@mail.gmail.com>
References: <4F90730D.1040808@trueblade.com>
	<CAKCKLWxO2v44Q8j+yG80aywtMtYC7BXFU9DHvnt1NV0-RPtmvQ@mail.gmail.com>
Message-ID: <CADiSq7e0cP38SThT_ugp4QudT8Q6vV5AMK=-N1A1-dEj4a3ucg@mail.gmail.com>

On Mon, Apr 23, 2012 at 9:51 AM, Michael Foord <fuzzyman at gmail.com> wrote:
> On 19 April 2012 21:18, Eric V. Smith <eric at trueblade.com> wrote:
>>
>> This reflects (I hope!) the discussions at PyCon. My plan is to produce
>> an implementation based on the importlib code, and then flush out pieces
>> of the PEP.
>>
>> In particular, I want to make sure the PEP addresses the various
>> objections that were raised, especially by Nick.
>>
>
> So a namespace package is a directory (tree) on sys.path. For a standard
> Python install how will these be installed?
>
> If you need to install "foo.bar" and "foo.baz" will distutils and packaging
> do the right thing? (And what specifically is the right thing for Python's
> own package management tools - merging the namespace packages or keeping
> them separate somehow?)

<lib_dir>/site-packages/foo/bar
<lib_dir>/site-packages/foo/baz

The whole point of dropping the __init__.py file requirement is that
merging the namespace portions becomes trivial, so you don't need to
worry about sys.path hackery in the normal case - you can just install
them into a common directory (adding it on install if it doesn't exist
yet, removing it on uninstall if the only remaining contents are the
__pycache__ subdirectory).

However, for zipfile distribution, or running from a source checkout,
you could instead provide them as <app_dir>/foo/bar and
<app_dir>/foo/baz and they would still be accessible as "foo.bar" and
"foo.baz". Basically, PEP 420 should mean that managing subpackages
and submodules becomes a *lot* more like managing top level packages
and modules.

Agreed the packaging implications should be specified clearly in the
PEP, though (especially the install/uninstall behaviour when namespace
portions get merged into a single directory).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia

From eric at trueblade.com  Wed Apr 25 14:48:59 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 25 Apr 2012 08:48:59 -0400
Subject: [Import-SIG] Implementation of PEP 420
Message-ID: <4F97F2BB.4080108@trueblade.com>

I have created http://bugs.python.org/issue14660 to track the
implementation of PEP 420. The code is checked in to features/pep-420.

The code currently works, but as described in the issue there's still
work to do.

For items related directly to the implementation, I think we should
discuss them on the issue. And continue the PEP discussion here.

Eric.

From carl at oddbird.net  Sat Apr 28 02:52:53 2012
From: carl at oddbird.net (Carl Meyer)
Date: Fri, 27 Apr 2012 18:52:53 -0600
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F90730D.1040808@trueblade.com>
References: <4F90730D.1040808@trueblade.com>
Message-ID: <4F9B3F65.2090406@oddbird.net>

On 04/19/2012 02:18 PM, Eric V. Smith wrote:
> This reflects (I hope!) the discussions at PyCon. My plan is to produce
> an implementation based on the importlib code, and then flush out pieces
> of the PEP.
>
> In particular, I want to make sure the PEP addresses the various
> objections that were raised, especially by Nick.

One clarity issue in the PEP:

"If the scan along the parent path completes without finding a module or 
package, then a namespace package is created."

This seems incomplete, and should say something like:

"If the scan along the parent path completes with finding a module or 
package, *but at least one directory was recorded,* then a namespace 
package is created."

The current wording seems to imply that any failed import would always 
cause the creation of a namespace package with an empty __path__, which 
I presume is not the intent.

Carl

From eric at trueblade.com  Sat Apr 28 12:27:19 2012
From: eric at trueblade.com (Eric V. Smith)
Date: Sat, 28 Apr 2012 06:27:19 -0400
Subject: [Import-SIG] PEP 420: Implicit Namespace Packages
In-Reply-To: <4F9B3F65.2090406@oddbird.net>
References: <4F90730D.1040808@trueblade.com> <4F9B3F65.2090406@oddbird.net>
Message-ID: <4F9BC607.20101@trueblade.com>

On 4/27/2012 8:52 PM, Carl Meyer wrote:
> On 04/19/2012 02:18 PM, Eric V. Smith wrote:
>> This reflects (I hope!) the discussions at PyCon. My plan is to produce
>> an implementation based on the importlib code, and then flush out pieces
>> of the PEP.
>>
>> In particular, I want to make sure the PEP addresses the various
>> objections that were raised, especially by Nick.
> 
> One clarity issue in the PEP:
> 
> "If the scan along the parent path completes without finding a module or
> package, then a namespace package is created."
> 
> This seems incomplete, and should say something like:
> 
> "If the scan along the parent path completes with finding a module or
> package, *but at least one directory was recorded,* then a namespace
> package is created."
> 
> The current wording seems to imply that any failed import would always
> cause the creation of a namespace package with an empty __path__, which
> I presume is not the intent.

Completely agree.  I changed "but" to "and", but otherwise used it
as-is. It's checked in.

Thanks!

Eric.