[Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"

[Glenn Linderman]

On 7/20/2011 6:05 AM, P.J. Eby wrote:
> At 02:24 AM 7/20/2011 -0700, Glenn Linderman wrote:
>> When I read about creating __path__ from sys.path, I immediately 
>> thought of the issue of programs that extend sys.path, and the above 
>> is the "workaround" for such programs.  but it requires such 
>> programs to do work, and there are a lot of such programs (I, a 
>> relative newbie, have had to write some).  As it turns out, I can't 
>> think of a situation where I have extended sys.path that would result 
>> in a problem for fancy namespace packages, because so far I've only 
>> written modules, not packages, and only modules are on the paths that 
>> I add to sys.path.  But that does not make for a general solution.
>
> Most programs extend sys.path in order to import things.  If those 
> things aren't yet imported, they don't have a __path__ yet, and so 
> don't need to be fixed.  Only programs that modify sys.path *after* 
> importing something that has a dynamic __path__ would need to do 
> anything about that.

Sure.  But there are a lot of things already imported by Python itself, 
and if this mechanism gets used in the stdlib, a program wouldn't know 
whether it is safe or not, to not bother with the 
pkgutil.extend_virtual_paths() call or not.

Plus, that requires importing pkgutil, which isn't necessarily done by 
every program that extends the sys.path ("import sys" is sufficient at 
present).

Plus, if some 3rd party packages are imported before sys.path is 
extended, the knowledge of how they are implement is required to make a 
choice about whether it is needed to import pkgutil and call 
extend_virtual_paths  or not.

So I am still left with my original question:

>
>> Is there some way to create a new __path__ that would reflect the 
>> fact that it has been dynamically created, rather than set from 
>> __init__.py, and then when it is referenced, calculate (and cache?) a 
>> new value of __path__ to actually search?
>
> That's what extend_virtual_paths() is for.  It updates the __path__ of 
> all currently-imported virtual packages.  Where before you wrote:
>
>      sys.path.append('foo')
>
> You would now write:
>
>      sys.path.append('foo')
>      pkgutil.extend_virtual_paths('foo')
>
> ...assuming you have virtual packages you've already imported.  If you 
> don't, there's no reason to call extend_virtual_paths().  But it 
> doesn't hurt anything if you call it unnecessarily, because it uses 
> sys.virtual_packages to find out what to update, and if you haven't 
> imported any virtual packages, there's nothing to update and the call 
> will be a quick no-op.

I think I would have to write

sys.path.append('foo')
import pkgutil
pkgutil.extend_virtual_paths('foo')

or I'd get an error.

And, in the absence of knowing (because I didn't write them) whether any 
of the packages I imported before extending sys.path are virtual 
packages or not, I would have to do this every time I extend sys.path.  
And so it becomes a burden on writing programs.

If the code is so boilerplate as you describe, should sys.path become an 
object that acts like a list, instead of a list, and have its append 
method automatically do the pkgutil.extend_virtual_paths for me?  Then I 
wouldn't have to worry about whether any of the packages I imported were 
virtual packages or not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20110720/c9adcddd/attachment.html>