[Python-ideas] Policy for altering sys.path

[Paul Moore]

On 7 May 2015 at 07:00, Thomas Güttler <guettliml at thomas-guettler.de> wrote:
>> Can you explain why?
>
> I forgot to explain the why I use a custom class. Sorry, here is the background.
>
> I want sys.path to ordered:
>
>  1. virtualenv
>  2. /usr/local/
>  3. /usr/lib
>
> We use virtualenvs with system site-packages.
>
> There are many places where sys.path gets altered.
>
> The last time we had sys.path problems I tried to write a test
> which checks that sys.path is the same for cron jobs and web requests.
> I failed. Too many places,  I could not find all the places
> and the conditions where sys.path got modified in a different way.

You do understand that by reordering sys.path like this you could
easily break code that adds entries to sys.path, by shadowing local
modules that the code is *deliberately* trying to put at the start of
the path?

I'm going to assume you have good reasons for doing this (and for
needing to - it seems to me that this is normally the order you'd get
by default). But even assuming that, I think your requirement is
specialised enough that you shouldn't be expecting other applications
to have to cater for it.

>> It seems pretty risky to expect that no
>> applications will replace sys.path. I understand that you're proposing
>> that we say that applications shouldn't do that - but just saying so
>> won't change the many applications already out there.
>
> Of course I know that if we agree on a policy, it wont' change existing code
> in one second. But if there is an official policy, you are able to
> write bug reports like this "Please alter sys.path according to the docs. See http://www.python.org/...."
>
> The next thing: If someone wants to add to sys.path, most of the
> time the developer inserts its new entries in the front of the list.

Generally, I would say that applications have every right to alter
sys.path to suit their needs. Libraries (typically) shouldn't alter
sys.path - in particular on import - without that being part of the
documented API.

If a library alters sys.path in a way that is a problem, and doesn't
document that it's doing so, then I think you have a case for a bug
report to that library. At a minimum they should document what they
do.

Your problem here is that pip is an *application* and so assumes the
right to alter sys.path. You seem to be using it as a library, and
that's where your problem lies. There *is* a reason we don't support
using pip as a library (this wasn't one we'd thought of, but the risk
of issues like this certainly was). With luck, now that you've brought
this point up, we'll remember if & when we do document a supported
pip-as-a-library API, and maybe deal with sys.path differently.

Paul

PS As I said before, it wouldn't be hard to fix the specific usage you
pointed out in pip, and I don't see a problem with submitting an issue
to that effect. Your custom subclass may still break pip, even after
we make such a change, but that'd be a separate issue with your
subclass, not a pip issue ;-) For this thread, though, I'm focusing on
your request for a "global policy".