[Python-3000] Changing the import machinery

[Ian Bicking]

Guido van Rossum wrote:
> I'm changing the list and the subject, pulling this quote out of python-dev:
> 
> On 4/20/06, in python-dev, Fredrik Lundh <fredrik at pythonware.com> wrote:
> 
>>I was hoping that for Python 3.0, we could get around to unkludge the
>>sys.path/meta_path/path_hooks/path_importer_cache big ball of hacks,
>>possibly by replacing sys.path with something a bit more intelligent than
>>a plain list.
> 
> 
> That's an excellent idea. Are there any volunteers here to help out?
> Even just listing specific use cases / scenarios that are currently
> difficult to solve right would be tremendously helpful. (I think that
> Phillip's and others' experience with setuptools might be very
> useful.)

Cleaning up import stuff would be excellent.  Time spent debugging 
imports is time wasted, but it happens all too often.

I would argue against any list of loaders, or list of anything.  That 
builds ambiguity directly into the system.  Without a list, if you want 
ambiguity, a container loader could search a list of loaders.  Or if you 
want to avoid all ambiguity, you could have a loader that was more picky.

Setuptools version-based eager loading can give you some confidence that 
everything you think you need is installed, but can't provide much 
confidence that everything you *think you are using* is actually what 
you are using.  That is, it's been fairly common in my own experience 
for me to realize some other version of a package is being loaded than 
what I thought, or I spend an inordinant amount of time tweaking 
requirements to get the right version of a package from one place 
without affecting another package that needs a different version (or 
perhaps is run with a different sys.path).

But, back to more concrete use cases:

* Right now it is pretty hard to set up an environment where changes 
elsewhere on the system can't leak in.  That is, installation of 
something in a system-wide site-packages can cause problems everywhere 
on the system, and even if you try to avoid these it is quite hard.  One 
strategy is setting up an entirely different environment (aka, a 
different prefix); this is heavy-feeling.  Another is avoiding site.py 
or using a custom site.py, but the tool support is iffy for that.

There's really just no good way to tell Python to leave well enough alone.

* Relatedly, installation and management when you don't have root or the 
cooperation of root can be hard.  I think the answer to this is much 
like the isolated environment, but the use case is fairly different.

* Configuration about where to install things (e.g., distutils.cfg) is 
separate from information about where to look for things (sys.path). 
These should form a consistent description of the environment, but 
currently they are disassociated from each other.

* Any kind of automatic installation is difficult, because you can't 
really count on being able to install even the most inoccuous package in 
an automated way.  There's too many manual overrides, and too many 
redundant options, and few people actually have their system set up to 
work without tweaking these options through the command line or other 
feedback.

* Personally I've settled on putting everything I make into a Python 
package that is distutils-installable.  But many people don't.  I'm not 
sure if this is just because the tools seem too hard, or the namespaces 
feel too deep, or all the documentation starts without using packages, 
or having '.' (sometimes) on sys.path does it, or what.  I'd rather 
there be consistent practices; but the consistent practices that we have 
that actually work (setup.py scripts and packages) are too heavy for a 
lot of people.

* People are seriously planning on using relative imports to manage 
their packages, and so an application will be 'installed' by putting it 
into another package.  Presumably unpacking it directly in some other 
package's directory.  Who knows what the version control plans are, or 
maintenance, or whatever.  I think it's a bad idea.  We need to give 
these people a carrot to keep them from doing this.

* Right now namespace packages are hard.  That is, a Python package 
(like 'zope') that is used by several distutils packages.  I almost feel 
like namespace packages should be installed flat, like 'zope-interface' 
and 'zope-tal', and turned into namespaces dynamically.

* The module layout is used both as an API and as an internal factoring 
of the code.  If you want to refactor the code you break the API. 
Personally I really like the strong connection between imports and code 
  location, and appreciate how easily I can find code as a result.  But 
setting up the scaffolding and warnings necessary when moving a module 
can be tiresome.

* Circular imports should fail more nicely.  Everyone suffers this at 
some time; maybe it can't be fixed, but at least it should be clear 
what's happening.

* You can't really tell if "from foo import bar" can be written as 
"import foo; bar = foo.bar", because it works if foo contains bar, but 
not if foo is a package and bar is a module in that package.



Well... I think that's maybe half way through the list of issues I have, 
but this email is already much too long.



-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org