[Import-SIG] Do we really need to read .pth files in PEP 382?

[Barry Warsaw]

On Jun 24, 2011, at 12:29 PM, P.J. Eby wrote:

>Do we really need to read .pth files in PEP 382?  If so, why?

It's an excellent question, Philip.  I know for the primary use case *I* care
about, the contents are really of no additional benefit.  And import.c does
carry a lot of extra complexity to read, parse, and combine them.  I suspect
that the vast majority of uses will have a single '*' in it and nothing more.

I also think the fact that they're almost, but not quite, like site.py .pth
files, share no common implementation in the core, but do share a file suffix
is confusing.

>In a common usecase for PEP 382 (large namespace packages like zope.*), there
>will be a long list of .pth files present, each of which contains only a '*',
>but which still must be opened and read by the implementation, while adding
>no new information.

And while we suspect that stat calls will dominate, opening, parsing, and
collating the file contents when it will almost always be just a '*' seems
wasteful.

Unlike traditional .pth files, lines starting with 'import' are not supported
in PEP 382.  That alone makes me uncomfortable with calling them `.pth`
files.  But like traditional .pth files, any line that the parser doesn't
recognize (i.e. that doesn't start with a # or is a single *) is taken as a
file system path, which itself must be stat'd before either being added or
discarded.  What's the point?

>However, if we had instead .ns files or .nspkg files or something like that,
>their mere *existence* could be construed as implying namespace-ness, and
>require no actual opens or reads.

+1.  Can we call YAGNI on anything else?  Certainly it would be easy enough to
extend the syntax of .ns files (or whatever color the bikeshed ends up getting
paint) should the need arise.  But if contents are meaningful from the start,
even if in practice it's never used, it'll be baggage we have to carry along
forever.

>If we separate the "this is a namespace" funtionality from "here are paths"
>functionality, ISTM that the "here are paths" functionality is already
>adequately met by the existing .pth machinery.  (As I'm not aware of any
>real-world use cases for pkgutils' .pkg files -- but perhaps someone can
>enlighten me on that?)
>
>Another consequence of this change is that it would simplify the PEP 302
>extension: instead of asking importers or loaders for a path, one could
>simply ask the importer whether a namespace exists, e.g.:
>
>    finder.namespace_exists(fullname)
>
>Returning either a subpath to put in __path__, or None if the named package
>is not a namespace.

While I support a simpler (and frankly, optional) additional API requirement
on PEP 302, I'd expect a function such as `namespace_exists()` to return a
pure boolean.  If it must return a subpath, then I'd prefer some other name.
Or maybe the __path__ could be passed in, allowing the finder do muck with it
however it wants.  Too dangerous?

>By comparison, the current proposal seems a bit overweight, considering that
>PEP 382 does not provide any use-case rationale for supporting anything
>besides '*' in .pkg files.  In fact, there doesn't seem to be any reason to
>put the '*' in the __path__ -- sys.namespace_packages suffices to indicate
>namespace-ness.  Code that wishes to extend existing namespace packages
>(e.g. setuptools) can simply perform the equivalent of the second loop above
>on any new path entries, for all entries in sys.namespace_packages.  (Well,
>not *all* entries, but all those that recursively yield reachable namespace
>additions.)
>
>Thoughts, anyone?

+1 for throwing away anything in PEP 382 we don't absolutely need.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/import-sig/attachments/20110624/497df93d/attachment.pgp>