[issue33944] Deprecate and remove pth files

[Barry A. Warsaw]

Barry A. Warsaw <barry at python.org> added the comment:

On Jul 5, 2018, at 14:23, Ivan Pozdeev <report at bugs.python.org> wrote:
> 
> Ivan Pozdeev <ivan_pozdeev at mail.ru> added the comment:
> 
>> They are very difficult to debug because they're processed too early.
> 
> .pth's are processed by site.py, so no more difficult than site/sitecustomize.
> You can e.g. run `site.addpackage(<dir>,<file>,None)' to debug the logic.

Not really.  By the time you have access to a REPL to run that, site.py has already run, so you already have an unclean environment.  Running with -S really isn’t feasible either since that’s often impossible (e.g. in a zip app like shiv or pex), or that leaves you with a broken environment so you can’t get to a usable REPL.  What you often have to do is actually modify Python to put a breakpoint in site.py to see what’s actually happening.  Yuck.

> 
>> They usually contain globs of inscrutable code.
> 
> An ability to contain code is there for a reason: to allow a module do something more intelligent than adding hardcoded paths if needed (e.g. pywin32 adds a subdir with .dll dependencies to PATH).
> 
> A chunk of code is limited to a single line -- a conscious limitation to deter misuse 'cuz search path setup code is presumed to be small.

Trust me, once you can execute arbitrary code in .pth files, you’re lost.  And packages *do* execute arbitrary code that is very difficult to debug.  And yes, those complex lines are both inscrutable and non-standard.

> If someone needs something other than path setup, they should do it upon the module's import instead.

Except they often don’t.

> If they insist on misusing the feature, Python's design does what it's supposed to do in such cases: "make right things easy, make wrong things hard”.

The problem comes when some random module you are including in your application does something weird in their .pth files that breaks assumptions *other* libraries or code is making.  It’s not as uncommon as it might seem.

> If there's a valid reason to allow larger code chunks, we can introduce a different syntax -- e.g. something like bash here-documents.

The size of the code chunks isn’t the only issue.  Running arbitrary code in a .pth file has all kinds of negative consequences.  It’s basically code that happens at import time, with all the problems that happen with that anti-pattern.

> 
>> Exceptions in pth files can get swallowed in some cases.
> 
> If this happens, it's a bug. A line from .pth is executed with "exec line", any exceptions should propagate up the stack as usual.
> 
>> They are loaded in indeterminate order.
> 
> Present a use case justifying a specific order.

Interdependent namespace packages.  If they get loaded in the wrong order, they can mess up __path__ settings, causing other namespace package portions to be un-importable.  Yes, this does happen!

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33944>
_______________________________________