[Python-Dev] Relative Package Imports

[Jim Fulton]

Guido van Rossum wrote:
> 
> Jim Fulton wrote:
> 
> > I wouldn't be in favor of making it more complicated if there wasn't
> > a good reason.  I think that, in working on the Zope framework,
> > I've found some pretty good reasons for relative imports.
> 
> And in a later message:
> 
> > I don't think the Package structure of Zope is flawed *except*
> > for the fact that it is one level too *shallow*. The ability to
> > do relative imports would be very helpful for the work we're doing.
> 
> But I haven't seen explained what it is that Zope is doing where
> relative packages would be helpful.

I posted an example in an earlier message.  I'll recast it here, 
hopefully more eloquently. :)  I'll also offer an alternate proposal
that also solves my (and I suspect, Marc-Andre's) problem.

Zope is an application platform.  It provides a mechanism for 
developers to plug their own products into Zope.  The idea is that
someone gets Zope from zope.org and installs it.  Then they get
third-party products from other places.

Zope products are python packages installed as sub-packages of the
Zope 'Products' packages.  Products are generally self-contained.  If
they need anything that's not part of standard Python or standard
Zope, they need to include it or install what they need in a
sub-package of another Zope package, 'Shared'.  

Because products come from "third parties", it is important that they
be self contained.  Making assumptions about the Zope or Python
environments or, worse, modifying the Zope or Python environments is a
bad idea.

In this context, consider the following concrete, though fictional
example.

Aaron has written a collection of modules that implement an RDBMS
system for Python, gadfly.  He also has a set of modules for parsing,
kjParsing, which is needed by gadfly.  Currently, these are just a
bunch of top-level modules distributed as a combined collection.  It
would make sense to turn these into two packages, gadfly and
kjParsing.  Now, if this was done, then the gadfly package would need
to use package imports for kjParsing modules, as in:

  import kjParsing.kjParser

So far, so good.  Now, suppose that someone wants to create a Zope
product, ZGadfly, that used gadfly.  The simplest approach would be to
include the gadfly and kjParsing packages in their Zope product.
Of course, this won't work, because the imports, like the one above,
will fail, because kjParsing is no longer a top-level package.

It wouldn't do any good to move gadlfy and kjParsing to the shared
package, although that might be desirable to share these packages
with other products.  They could try to stuff the packages into the
Zope or Python paths, but that would break the rules and lead to
problems in the long term.

Hopefully, this illustrates the problem.  I think that this will be a
common problem in the future, as people build bigger and bigger
systems with Python that reuse other people's packages.

I'd be curious to hear how folks would solve this problem.

Personally, I'd like the problem to go away. :)  I'd like the Python
import rules to change to make this solvable without import hooks or
path hacking.  I can think of two ways to approach this:

  - Relative parent imports:

      import __.kjParsing.kjParser

    Note that Python already supports relative imports for 
    going down.  For example, we use gadfly and kjParsing together
    as a single sub-package of our ZGadflyDA product.

  - Gradually less local searches.

    Currently, when doing an import in a package, two paths are
    searched, the package path and then the Python path.  If there are
    intermediate packages, then perhaps their paths should be searched
    as well.

    For example, suppose we have the directory structure:

      Products/ZGadfly/gadfly
                       kjParsing

    where Products is a top-level package, and we did an import in a
    module in gadfly:

      import kjParsing.kjParser

    Python would search the path of the package
    Products.ZGadfly.gadfly first, as it does now.  This search would
    fail. Then it would search the path of Products.ZGadfly, where it
    would find kjParsing, and the import would succeed.

    This approach has the benefits:

      o It solves the problem. :)

      o It has no impact on un-nested packages,

      o It requires no code changes; it doesn't use the
        ugly __.

Thoughts?

Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org    

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission.  Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.