Incompatible idioms: relative imports, top-level program file (was: Setuptools, __init__ and __main__)

[Ben Finney]

Ethan Furman <ethan at stoneleaf.us> writes:

> On 02/06/2015 02:56 PM, Ben Finney wrote:
> > It is a deliberate design decision that direct import of a module
> > makes that module blind to its location in the package hierarchy.
> > 
> > That's a design decision I deplore, because it makes something that
> > should be easy (write a command directly into a file and invoke that
> > file as an executable program) tortuously difficult when the program
> > comprises several modules.
>
> Can you explain that a bit more?

A program will often have enough complexity that its implementation
occupies several sub-modules. There's no need to explose those in a site
package, they normally only need to be local to the application.

So the correct idiom is ‘from __future__ import absolute_import’ and
keep the application's own implementation imported via relative imports
in the same package: ‘from . import implementation_detail’.

But relative import is not possible when the top-level module is
executed by the normal ‘fooprog’ usage. The top-level file is named
‘fooprog’ and is marked executable, so that it can be run directly as
soon as it's written.

Python deliberately divorces the top-level module from its package, so
it can't access its own relative modules! The relative import fails with
an ImportError “Attempted relative import in non-package”.

<URL:https://mail.python.org/pipermail/python-list/2014-November/694385.html>

So the two correct idioms – relative import for non-public modules, and
run a file directly as a command – are explicitly thwarted by Python's
import behaviour when you try to use them together.

> Surely if the writing a command into a file was going to be easy, then
> so would be writing it to __main__.py instead?

With Bash shell, or Perl, or Ruby, or countless other languages, I put
the top-level code in the *exact same file* that I'm going to execute,
named ‘fooprog’. No additional infrastructure files unless my program
architecture indicates they'll help.

Python's refusal to allow this leads to roundabout hacks like “make a
sub-directory ‘fooprog/’, write the code into ‘__main__.py’”, or “run
the program by typing ‘python -m fooprog’”, or “write a ‘./setup.py’
configuration and dicker around with Distutils's rules and install it
before you can run your program”.

A system needing baroque hacks like this doesn't really deserve to be
characterised as a “scripting language”, IMO. Fortunately, I don't like
that term anyway (we already have the term “program” for what Python
calls scripts) so avoid characterising Python that way for other reasons
too :-)

-- 
 \        “Sane people have an appropriate perspective on the relative |
  `\     importance of foodstuffs and human beings. Crazy people can't |
_o__)                 tell the difference.” —Paul Z. Myers, 2010-04-18 |
Ben Finney