Proposal for a modified import mechanism.

[Frederic Giacometti]

eric wrote:

> I have to agree with Prabhu on this one.  The current behavior of import,
> while fine for standard modules and even simple packages with a single
> level, is sub-optimal for packages that contain sub-packages.  The proposed
> behavior solves the problem.
>
> Handling the packaging issues in SciPy was difficult, and even resulted in a
> (not always popular) decision to build and overwrite the Numeric package on
> machines that install SciPy.  Prabhu's import doesn't resolve all the issues
> (I think packages may just be difficult...), but it would have solved this
> one.  The proposed import allows us to put our own version of Numeric in the
> top SciPy directory.  Then all SciPy sub-packages would grab this one
> instead of an existing site-packages/Numeric.

But then, this is not an import problem.
If you use Numeric, you call Numeric. If you call something other than Numeric,
just give a different name, and all the confusion will go away.
If you're worried that you've already encoded the Numeric name 50 times  into
300 files; run a python script over these 300 files; this will do the renaming
of the 15.000 occurences of the Numeric name.

> That makes SciPy
> self-contained and allows people to try it out without worrying that it
> might break their current installation.  There are other solutions to this
> problem, but Prabhu's fix is by far the easiest and most robust.

And then, in maintenance/integration phase, sometimes 'Numeric' will call
Numeric, some other times it will your package ?

What if somebody, for some reason I know nothing of (e.g. probably some
integration) wants to call Numeric and your Numeric package in the same module ?
Wish them tough luck to sort out this poisoned gift....

> Prabhu's import also has some other nice benefits.  Some of the sub-packages
> in SciPy are useful outside of SciPy.  Also sometimes it is easier to
> develop a packages outside of the SciPy framework.  It would be nice to be
> able to develop a module or package 'foo' outside of SciPy and then move it
> into SciPy at a later date.  However, every SciPy sub-package that referred
> to foo prior to its inclusion in SciPy now has to be updated from 'import
> foo' to 'import scipy.foo'.  These kind of issues make it very painful and
> time consuming to rearrange package structures or move modules and
> sub-packages in and out of the package.

There are basic python scripts which do this painlessly. If you're really
working on a large project, there's a project architect which normally would
take care of such things, and for whom this should not be a too much of a
problem.


> Simplifying this will improves
> package development.
>
> > I'm personnally against anything that enlarges the search path uselessly;
>
> Hopefully I've explained why it is useful for complex packages.

Python helps in many areas, but expecting it to palliate for the package design
and architecture flaws that inexorably surface anytimes something non-trivial is
developped, might be somehow at the edge. Python has not yet replaced the need
for relevant software architects.

>
> > because the obvious reason of increased name space collision, increased
> > run-time overhead etc...
>
> I'm missing something here because I don't understand why this increases
> name space collision.  If the objection is to the fact that SciPy can have a
> version of Numeric in it that masks a Numeric installed in site-packages, I
> guess I consider this a feature, not a bug.

Actually, it is normally worse than a bug: it is a source of bug tomorrow in
your application - of all the bugs you'll have when your programmer will be
confusing the two Numeric packages, as well as all the mainteance and
integration problems you'll have down the line -.

But by then, hopefully for you, you'll be somewhere else... The sad reality of
most projects :((

>  Afterall, this is already the
> behavior for single level packages, extending it to multi-level packages
> seems natural.  If this isn't your objection, please explain.
>
> The current runtime overhead isn't so bad.  Prabhu sent me a few numbers on
> the SciPy import (which contains maybe 10-15 nested packages).  I attached
> them below -- the overhead is less than 10%.  It should be negligible for
> standard modules as only packages are really affected (right Prabhu?).

And that's how, when you cumulate of the overheads for all new features, you get
potenially +100-200% overhead on the new releases.
Albeit all the efforts of the Python team, Python 2.0 is up to 70% slower than
python 1.5.2; Python 2.1.1 is up to 30% slower than python 2.0, and so on...
So, +10% on only such a minor features is anything but negligible :(((

FG