[Python-ideas] Pseudo-package for current directory

[Chris Angelico]

On Tue, Mar 8, 2016 at 5:04 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Tue, Mar 08, 2016 at 03:54:41AM +1100, Chris Angelico wrote:
>> Every once in a while, the issue pops up again of "import X" picking
>> up X.py from the current directory, when a stdlib or pip-installed
>> module named X was wanted.
>
> Couldn't we (almost) fix that in an almost backwards-compatible way by
> moving the current directory to the end of the path, instead of the
> beginning?

Good question. Fortunately that one can be tested without major
language hacking. I'm curious to know the answer.

>> Python almost has a mechanism for
>> protecting against this, in the form of explicit package relative
>> imports; the only problem is that the current directory isn't a
>> package.
>
> We shouldn't be talking about "current directory", since that's only
> relevant when you run Python without specifying a script to run. It is
> actually the directory containing the script being executed, not the
> current directory, which gets added to the path. If there is no script,
> the faux path '' is added to the path, and that gets treated as if it
> were '.' (the current directory).

Fair criticism. What I really mean is "script directory", which
happens to align in the specific (albeit common) case of "python3
script.py" with no path on it. The rest of my proposal is the same,
though.

>> So the solution is to treat the current directory as a pseudo-package.
>> It'd be a backward-incompatible change, so it would need to be
>> explicitly invoked. Something like:
>>
>> python3 -p somefile.py
>>
>> which would pretend to create an __init__.py in the current directory,
>> change to the parent, and "from dirname import somefile".
>
> Seems awfully convoluted and hard to understand.

I know; that's the result of being pedantic. Here's an alternative description:

$ python3 -p somefile.py

pretends that the current directory is a package, and imports somefile
from that package.

> How will it interact with running real packages? How about running
> modules using -m?

I don't think it would be any different, but haven't tested. They'd be
exactly the same thing, except that "python3 -m X.Y" looks for Y.py
inside package X on sys.path, and "python3 -p somefile.py" takes a
file name (which might include a path), and runs it directly. If
you're running a real package that isn't on sys.path, you can either
put it on sys.path, or use -p mode. Something like this, which
currently works (and would still work if the current directory were
removed from sys.path):

$ mkdir pkg
$ echo 'print("init")' >pkg/__init__.py
$ echo 'print("module")' >pkg/module.py
$ PYTHONPATH=. python3 -m pkg.module
init
module

Or alternatively, this:
$ python3 -p pkg/module.py

If the semantics of -p are "pretend that __init__.py exists", without
precluding one that really does, this should behave correctly.

> Even if it works, it relies on the user being knowledgeable enough to
> run `python3 -p script` instead of `python3 script`, which means the
> people who most need this new feature are the least likely to use it. I
> think that's the fatal objection to this: if the user knows enough to
> run -p, she knows enough to diagnose an accidental shadowing.

Right, this is a concern. But I don't want to break backward
compatibility. Otherwise, the common situation of having multiple .py
files in the same directory and having them import each other would be
broken. Ultimately, I'd like to offer this protection by default. But
I'm seeing this as similar to reminding people to type "python3"
instead of just "python"; now we just teach people to type "python3
-p", and if they happen to miss off the 3, the system will now tell
them (because "python2 -p" is an error).

If it's done as an environment variable as well or instead, would that
be better? Python distributors and educators could choose to activate
it by default, and a venv could apply the protection.

>> Idle could be protected from accidentally importing someone's
>> "html.py", because "import html" shouldn't ever import from the
>> current directory.
>
> That won't help when the user runs
>
> python3 html.py
>
> (well perhaps it will help *specifically* with IDLE, but not with the
> general problem of accidental shadowing).

True, but if you're asking for help and you're running a thing called
"html.py" or "complex.py", the people you talk to have a chance of
noticing. But the presence of such a file in the current directory
could mess stuff up, and a lot of people don't think to mention every
other file they've ever worked on (in their "Python Tinkering"
directory) when they run into difficulties.

ChrisA