PyWart: "Python's import statement and the history of external dependencies"

[Rick Johnson]

Python's attempt to solve the "external dependencies problem"
has yet to produce the results that many people, including
myself, would like.

Actually, Python is not alone in this deficiency, no, Python
is just *ANOTHER* language in a *STRING* of languages over
the years who has *YET AGAIN* implemented the same old
crusty design patterns, packaged them in a shiny metallic
wrapping paper with a big red bow on top, and hoped that no
one would notice the stench...

    WELL I NOTICED M.F.!

Before we can formulate a solution to this mess, we must
first obtain an "inside perspective" of the "world" in which a
Python script lives during run-time.

============================================================
 Welcome to "Python Gardens": a master planned community
============================================================

Well... urm, sort of. @_@

I like to analyze problems, when possible, in relation to
real world "tangible" examples. So for me, i like to think
of the main script (aka: __main__) as an apartment building,
and each module that runs under the main script as a single
apartment, and finally, the inhabitants of the apartments as
objects.

Furthermore, we can define the apartment building as being
"global space", and each individual apartment as being
"local space". The first law we encounter is that "global
space" is reserved for all tenants/guest, but "local space"
is by *invitation* only!

    You can think of "import" as similar to sending out an
    invitation, and requesting that a friend join you inside
    your apartment (we'll get back to "import" later).

And of course, with this being the 21st century and all, every
apartment has *local* access to *global* resources like
water and electrical.

    In Python, these "global resources" are available
    "locally" in every module via the implicit namespace of
    "__builtins__". You can think of built-ins as a "house
    keeper" robot that lives in every apartment (I call mine
    "Rosie"). It was there when you moved in --a "house
    warming" gift of sorts-- and it helps you with chores
    and makes your life a little easier. Sometimes i abuse 
    my robot but i always apologize afterwards.
    
Now that we know how the global and local spaces are defined
(in the context of modules), and what implicit/explicit
"gifts" (are supplied), and "rules" (are demanded) in our
little Python world, we have a good basis to start
understanding why Python's import mechanism is at best a
"helpful failure" that only *naively* attempts to
streamline the management of such an important language
feature as external dependencies!

Up until this point, our apartment example has not mapped
the actual *mechanics* of "import" to a real world example,
but now that we have the correct "perspective", we can tread
into the dark and damp underworld that is "import".

Remember when i said: "import is like sending out an
invitation"? Well, in actuality, import is only
*superficially* similar to "sending out an invitation".

You see, when you send an invitation in real life, the most
important two things you'll need are a *name* and an
*address* --a name so you'll know *who* to send the
invitation to, and an address so you'll know *where* to send
the invitation-- but Python's import mechanism does not work
that way. When you import an external dependency into a
Python module all you need is the name of a module --
addresses are neither required *OR* allowed!

Like almost all modern languages, Python has adopted the
ubiquitous practice of internalizing a mapping of known
directories from which to search for modules (called a
search path), so when we "import somemodule" the following
happens

(Note: I'm not going into too many details for the sake of topic):

    1. Python checks if the external resource has already been loaded

    if not 1, then

    2. Python looks in a few "special places"

    if not 2, then

    3. Python searches one-by-one the directories in sys.path

RESULT: Python finds the resource or barfs an exception.

I have no problem with step 1, however, step 2 and step 3
can be redundant, excessive, and even unreliable. I can
explain better why i levee such harsh words for "import" by
going back to our apartment building example.

Let's imagine that Python is the "lobby boy" of our apartment
building. And in our little example, one of the duties of
the "lobby boy" is to manage invitations between tenants
and guests. Each time a tenant wants to invite someone into
their apartment, they must pick up the phone, press the
import button (which connects them to the lobby boy via a
voice connection) and they say the name of the person they
want to invite.

    But *EVEN IF* they know the address of the recipient, they
    are not allowed to say the address to the lobby boy, no,
    because *THIS* building is owned by a evil tyrant,
    who has declared that any mention of an address when
    calling import is punishable by the fatal exception!

So, being a good tenant who does not want to *DIE*, they
simply mention the recipients name, roll their eyes, hang up
the phone, and hope that lobby boy can locate the correct
recipient

It is at this point that the lobby boy (aka: Python) springs
into action. The lobby boy has a gargantuan task ahead of
him, he must locate a tenant based solely on the name with
no idea of which apartment the tenant resides. And,
realizing that he has a terrible memory, and that some of
the building's pranksters have sent him on "wild goose chases"
for recipients that do not exist, his first action is to
check his daily log and ensure he did not *already* deliver
the invitation...

    DAMN! Not in the log!

So then the lobby boy checks for a few special places...

    DAMN! not in my "special places" either!

Sadly, the lobby knows that now he must check the ENTIRE
building, of which contains thousands of tenants. So he
starts on floor one (sys.path[0]), knocks on all the doors,
then proceeds to floor two (sys.path[1]), knocks on all
those doors..., and so on and so forth until he locates the
recipient or dies of exhaustion -- whichever comes first.

It is this point that i find the import machinery to be
lacking. It's not only the *needless* exhaustive searching
via such an unintelligent algorithm, but also the great
possibility of importing the wrong resource.

    "Yeah Rick, but who wants to include a file-path for
    importing modules, the whole reason for import was to
    abstract away the file-system and file-paths, etc... Are
    you proposing we go back and re-adopt the historical
    conventions?"

Absolutely not. In most day to day usage Python's import
statement works just fine, it's a cycle burner, but who
cares about a lobby boy anyway right?

I have thousands of modules on my hdrive, and many many
directories holding them, even if i could overlook the
"cycle bonfire" of import's underlying machinery, the list of
intuit-able names for modules are finite, and i'm not about
to start tagging file-names because then i'm back to the
file-system again -- but instead of slashes i'm using
underscores.

    NO THANKS!.

What the creators of "import" fail to understand is that a
trade-off has been made, and the Python programmer has been
given the short end of a very long stick, and now we're
being beaten with the very long stick and he nothing to 
defend himself with but a tooth-pick! 

Creating an "implicit name resolution system" (aka: import) 
to abstract away an "explicit name resolution system" 
(file-paths) has resulted in more problems that it can solve:

    1. Name clashes!
    2. Smaller name pool!
    3. Machinery is too implicit!
    4. Circular imports are inevitable!
    5. Much too difficult to use and/or explain!
    6. Too many "gotchas"!

Of course, if "import" is not problematic enough for you,
one you use one of the many modules like importlib, imp,
importil, ihooks, or __import__ function, and then become
more confused!! -- Python's external dependency is not only
broken, it's discombobulated and downright confusing!

In closing, there is only one thing you need to know about
Python's import statement:  it is deceptively easy to
underestimate it's ability to *EFF UP* your expectations!