PyWart: "Python's import statement and the history of external dependencies"

[Rick Johnson]

On Friday, November 21, 2014 9:34:55 AM UTC-6, Ian wrote:
> On Thu, Nov 20, 2014 at 8:53 PM, Rick Johnson
> > FOR INSTANCE: Let's say i write a module that presents a
> > reusable GUI calendar widget, and then i name the module
> > "calender.py".
> >
> > Then Later, when i try to import *MY* GUI widget named
> > "calendar", i will not get *MY* calendar widget, no, i will
> > get the Python calendar module.
>
> Because like a fool you created a local module with a
> totally generic name like "calendar" and dumped it into
> the top-level namespace.

Are you also going to call drivers "fools" because they bought
a "certain brand" of car only to have the airbag explode in
their face? What's next, are you going to blame them for not
wearing a mask? Why am *i* the fool when it's obvious that
the creators of Python were either shortsighted and/or
careless with the designs? The only people who suffer are
those who put their trust in the designer, and not the
designer himself -- something is wrong with this picture!

> The Python import system gives you the ability to create
> packages, so use them -- create a package to contain all
> the widgets you create. If they're only for your own use,
> then you can just call it ricklib or anything else you
> want.

Of course, I did that a long time ago! But like everything
in Python, when your try to be cleaver and create a
workaround for the design flaws of this language (and there
are many!) you end up with shards of jagged metal stuck in
your face!

    GvR MUST BE A HUGE FAN OF THE JIGSAW MOVIES!

> Now you can drop as much stuff in there as you like, and
> none of it will ever conflict with the standard library
> (unless a standard "ricklib" module is added, which is
> unlikely).

Yes, and now we've solved one problem by replacing it with
it's inverse -- try importing the *python lib* calendar
module and all you will get is your local "intra-package"
version. Now, the only way to get to the lib module is by
mutilating sys.path, or using an import utility module to
"import by filepath".

    THANKS PYTHON!

You are correct though, Python's packaging system CAN BE
used to isolate "intra-package" scripts from clashing with
outside scripts, HOWEVER, there are many caveats that one
must know (which are NOT intuitive!) to use Python's
packaging system without requiring a plastic surgeon on
retainer!

Lets say i take your advice, and i want to use python
packages to protect a set of modules from the "global import
scope".

############################################################
#                     Explosive Trap 1                     #
############################################################
# Any attempt to import a stdlib module, who's name        #
# matches one of the modules within the current package,   #
# will result in importing the local "package module",     #
# forcing you to: (1) inject search paths into sys.path    #
# manually (2) import a module to correctly import the     #
# dependency                                               #
############################################################

YUCK!

############################################################
#                     Explosive Trap 2                     #
############################################################
# Any code in the __init__ file will be executed NOT ONLY  #
# when the package is imported DIRECTLY, but even when the #
# package name is referenced as part of a larger "absolute #
# import path"                                             #
############################################################

OUCH!

Anyone would expect that when *DIRECTLY* importing a
package, if the __init__ file has code, then THAT code
should be executed, HOWEVER,  not many would expect that
merely "referencing" the package name (in order to import a
more deeply nested package) would cause ALL the
intermediate __init__ files to execute -- this is madness,
and it prevents using an __init__ file as an "import hub"
(without side-effects)!

    "Yeah rick, but why would you want to use an __init__ file
     as an "import hub"

Because the alternative is messy. If i have a collection of
modules under a package, sometimes i would like to import
all the "exportable objects" into the __init__ file and use
the package as an "import hub". Imagine a package layout
like this:

    + ricklib
      __init__.py
      + subpkg1 (ricklib.subpkg1)
        __init__.py
        module1.py
        module2.py
        module3.py
        + subpkg1a (ricklib.subpkg1.subpkg1a)

And the contents of "ricklib.subpkg1.__init__.py" are as
follows:

    from module1 import Class1
    from module2 import Class2
    from module3 import Class3

Now, when i want to import a number of modules from subpkg1
i can do:

    from ricklib.subpkg1 import Class1, Class2, ...

    Instead of this:
    
    from ricklib.subpkg1.module1 import Class1
    from ricklib.subpkg1.module2 import Class2

And everything will work fine, UNTIL, i try to access
"subpkg1a"

    from ricklib.subpkg1.subpkg1a import something

By simply MENTIONING the name "subpkga" (within the
"absolute path" of "ricklib.subpkg1.subpkg1a"), Python will
execute the "ricklib.subpkg1.__init__.py" file, thereby
burning cycles importing things that don't even need at the
time! And god forbid you have some "print statements" in
there! 

    BETTER GO MAKE A SANDWICH WHILE THIS ONE LOADS!

> > This is another confusing fundamental of Python, it seems *MORE*
> > intuitive that changes to the "import search path" would only
> > affect the *CURRENT* module's "import search path", but that's
> > is not the case!
>
> This just seems like it would create a maintenance
> nightmare. Before you could import anything, you'd have to
> figure out what the search path is for the particular
> module you're working and whether it happens to include
> the particular package you want. You find it doesn't, so
> you make a local change to the search path. Later, you
> make the same change to other modules that need to import
> it. Later still, the package moves to a different location
> in the file system, and now you get to go through and
> update every one of those modules with the new directory.

But the current "global import search path" injections are
just the inverse. You make changes to sys.path in one
module, and if you fail to reset the changes before
execution moves to the next module in the "import chain",
then that module's import search path will be affected in
implicit ways that could result in importing the wrong
module.

At least with my proposed "local import search mutations",
the modifications to sys.path would NOT propagate outwards,
because each module would recieve a fresh copy of sys.path
from which to use "as-is" or to modify "as needed", but
those changes would only be relevant within the current
iteration of the import chain.

I think you fail to appreciate my proposal because you are
forgetting the importance of maintaining a "linear
structure" for external dependencies. This "idea" that a
script who is "higher" in the import chain, should be
affecting the "import search paths" defined in the all the
"lower scripts", is doomed to failure, and it will fail
hard.

> > That's ridiculously noisy. I have an idea, if you *REALLY*
> > want to introduce boilerplate hell then Python should adopt
> > "import header files", NOT!
>
> It's three lines of code to replace one. Two if you exclude the
> importlib.machinery import that doesn't need to be repeated.  Is this
> really any worse than something like:
>
> local_search_path.insert(0, "/path/to/local/module")
> import my_local_module
>
> that you are proposing?

If the changes were LOCAL, then i would have no problem to
this type of mutation, But they are not.