[Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen

[Terry Reedy]

On 11/1/2015 1:06 AM, Steven D'Aprano wrote:
> CC'ing Python-Ideas. Follow-ups to Python-Ideas please.
>
> On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote:
>
>> Leaving IDLE aside, the reason '' is added to sys.path is so that people
>> can import their own modules.  This is very useful.  Shadowing is the
>> result of putting it at the front.  I have long thought this a dubious
>> choice.  If '' were instead appended, people could still import modules
>> that did not duplicate stdlib names.  Anyone who wanted shadowing could
>> move '' to the front.  But then shadowing would be intentional, not an
>> accident.
>
> Terry is right. Shadowing should be possible, and it should require a
> deliberate decision on the part of the programmer.
>
> Consider the shell, say, bash or similar. My understanding is that the
> shell PATH deliberately excludes the current directory because of the
> possibility of malicious software shadowing usual commands in /bin etc.
> If you want to run an executable in the current directory, you have to
> explicitly provide the path to it: ./myscript rather than just myscript.
>
> Now Python isn't exactly the shell, and so I'm not proposing that Python
> does the same thing. But surely we can agree on the following?
>
> - Shadowing explicitly installed packages, including the stdlib, is
>    *occasionally* useful.
>
> - But when shadowing occurs, it is *nearly always* accidental.
>
> - Such accidental shadowing often causes problems.
>
> - And further more, debugging shadowing problems is sometimes tricky
>    even for experienced coders, and almost impossible for beginners.
>
>    (It's not until you've been burned once or thrice by shadowing that
>    you recognise the symptoms, at which point it is then usually easy to
>    debug.)
>
> - Hence, we should put the onus on those who want to shadow installed
>    packages) to do so *explicitly*, or at least make it easier to avoid
>    accidental shadowing.
>
>
> I propose the following two changes:
>
>
> (1) Beginning with Python 3.6, the default is that the current directory
> is put at the end of sys.path rather than the beginning. Instead of:
>
>      >>> print(sys.path)
>      ['', '/this', '/that', '/another']
>
> we will have this instead:
>
>      >>> print(sys.path)
>      ['/this', '/that', '/another', '']
>
> Those who don't shadow installed packages won't notice any
> difference.

Serhiy pointed out that code that unconditionally executes 
sys.path.pop(0) to avoid shadowing of its stdlib imports might break if 
it imports from what is now sys.path[1].  This currently (2.7, 3.4, 3.5) 
is <somewhere>pythonxy.zip.  (In 2.7 and 3.4 on Windows, somewhere = 
/windows/system32, In 3.5, the install directory). I say 'might' because 
I don't know if the presence of pythonxy.zip always means that 
<installdir>/lib is absent. If it is gone, the imports will fail.

Since use of zip is rare, the transition might be doable.  Adding "if 
sys.path[0] == '':" is backwark compatible.  If the import system had 
available the set of /lib modules, as I proposed on python-list, then 
the error handler could diagnose the situation of a lib module import 
failing because of an existing pythonxy.zip not being on sys.path. I 
believe the set would also help for generating deprecation messages.

> Scripts which deliberately or unintentionally shadow installed packages
> will break from this change. I don't have a problem with this. You can't
> fix harmful behaviour without breaking code that depends on that harmful
> behaviour. Additionally, I expect that those who rely on the current
> behaviour will be in a small minority, much fewer than those who will be
> bitten by accidental shadowing into the indefinite future. And if you
> want the old behaviour back, it is easy to do so, by changing the path
> before doing your imports:
>
>      import sys
>      if sys.path[-1] == "":  sys.path = [""] + sys.path[:-1]
>
> or equivalent.
>
> I do not belive that it is onerous for those who want shadowing to have
> to take steps to do so explicitly.

The question is whether it would be too onerous for those explicitly 
avoiding shadowing now to have to change their explicit code.

 > That can be added to your scripts on
> a case-by-case basis, or your PYTHONSTARTUP file, by modifying your
> site.py, or (I think) by putting the code into the sitecustomize or
> usercustomize modules.
>
> (2) IDLE doesn't need to wait for Python 3.6 to make this change. I
> believe that IDLE is permitted to make backwards incompatible changes in
> minor releases, so there is no reason why it can't change the path
> effective immediately.

IDLE can change how it operates with its own code.  I plan to add a 
conditional pop before IDLE does its own imports.  However, IDLE is not 
free to change how user code is executed, except to better imitate 
CPython than it does now. Hence '' will have to be put back for user 
code execution.

-- 
Terry Jan Reedy