[Python-Dev] PEP 382: Namespace Packages

[M.-A. Lemburg]

[Resent due to a python.org mail server problem]

On 2009-04-03 22:07, Martin v. Löwis wrote:
>> I'd like to extend the proposal to Python 2.7 and later.
> 
> I don't object, but I also don't want to propose this, so
> I added it to the discussion.
> 
> My (and perhaps other people's) concern is that 2.7 might
> well be the last release of the 2.x series. If so, adding
> this feature to it would make 2.7 an odd special case for
> users and providers of third party tools.

I certainly hope that we'll see more useful features backported
from 3.x to the 2.x series or forward ported from 2.x to 3.x
(depending on what the core developer preferences are).

Regarding this particular PEP, it is well possible to implement
an importer that provides the functionality for Python 2.3-2.7
versions, so it doesn't have to be an odd special case.

>> That's going to slow down Python package detection a lot - you'd
>> replace an O(1) test with an O(n) scan.
> 
> I question that claim. In traditional Unix systems, the file system
> driver performs a linear search of the directory, so it's rather
> O(n)-in-kernel vs. O(n)-in-Python. Even for advanced file systems,
> you need at least O(log n) to determine whether a specific file is
> in a directory. For all practical purposes, the package directory
> will fit in a single disk block (containing a single .pkg file, and
> one or few subpackages), making listdir complete as fast as stat.

On second thought, you're right, it won't be that costly. It
requires an os.listdir() scan due to the wildcard approach and in
some cases, such a scan may not be possible, e.g. when using
frozen packages. Indeed, the freeze mechanism would not even add
the .pkg files - it only handles .py file content.

The same is true for distutils, MANIFEST generators and other
installer mechanisms - it would have to learn to package
the .pkg files along with the Python files.

Another problem with the .pkg file approach is that the file extension
is already in use for e.g. Mac OS X installers.

You don't have those issues with the __pkg__.py file approach
I suggested.

>> Wouldn't it be better to stick with a simpler approach and look for
>> "__pkg__.py" files to detect namespace packages using that O(1) check ?
> 
> Again - this wouldn't be O(1). More importantly, it breaks system
> packages, which now again have to deal with the conflicting file names
> if they want to install all portions into a single location.

True, but since that means changing the package infrastructure, I think
it's fair to ask distributors who want to use that approach to also take
care of looking into the __pkg__.py files and merging them if
necessary.

Most of the time the __pkg__.py files will be empty, so that's not
really much to ask for.

>> This would also avoid any issues you'd otherwise run into if you want
>> to maintain this scheme in an importer that doesn't have access to a list
>> of files in a package directory, but is well capable for the checking
>> the existence of a file.
> 
> Do you have a specific mechanism in mind?

Yes: frozen modules and imports straight from a web resource.

The .pkg file approach requires a directory scan and additional
support from all importers.

The __pkg__.py approach I suggested can use existing importers
without modifications by checking for the existence of such
a Python module in an importer managed resource.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2009-03-19: Released mxODBC.Connect 1.0.1      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/