[Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)

[VanL]

Hi Paul,

To start with, I appreciate your comments, and it is worth having both 
sides expressed.

On 3/22/2012 12:59 PM, Paul Moore wrote:
> I'm repeating myself here after I promised not to. My apologies, but I 
> don't think this posting captures the debate completely. One reason I 
> suggested a PEP is to better ensure that the arguments both pro and 
> con were captured, as that is a key part of the PEP process. 

I would be happy to write up a PEP.
> First of all, this difference is almost entirely *invisible*. Apart
> from possibly setting PATH (once!) users should not be digging around
> in the Python installation directory. Certainly on Windows, it is a
> very unusual application that expects users to even care how the
> application is laid out internally. And I suspect that is also true on
> Unix and Mac.

This is a good point; it is mostly visible in the virtualenvs. If it 
only changed in virtualenvs, I would be happy. The policy, though, is 
that the virtualenv follows the platform policy.

> Secondly, the layouts are not as similar as you claim here, if I
> understand things correctly. on Unix Python is installed in
> /usr/local/bin so there isn't even a "Python installation directory"
> there. And Macs use some sort of Mac-specific bundle technology as I
> understand it. To be honest, I don't think that there's a lot of
> similarity even with your proposed changes.

I was summarizing here because, frankly, there are hardly any OS/2 
users, so it would be mostly Windows users affected by this change. Also 
as noted, I suggest that all platforms standardize on bin, lib, and 
include, just as I laid out.

That said, while I think that the above is a good idea, my personal 
ambitions are more modest: If the names of the top-level directories 
only were changed to 'bin', 'lib', and 'include' - never mind 
differences under 'lib' - I would be happy. In fact, even the one change 
of 'Scripts' to 'bin' everywhere would get 90% of my uses.

> The issue with virtualenv and pyvenv may be more significant. But
> you're only mentioning those incidentally.

I am approaching it from the platform level because of the policy that 
virtualenvs match the platform install layout. If instead virtualenv 
layouts were standardized, that would end up making me just as happy.


> (I also get a sense of undue haste -
> "we can change the Python layout for 3.3, but changing packaging and
> virtualenv is a much longer process"...)

Honestly, I didn't expect that much resistance. None of the people I 
talked to in person even cared, or if they did, they thought that 
consistency was a benefit. But now that virtualenvs are going in in 3.3, 
I see this as the last good chance to change this.


> This presupposes that your development workflow - developing in place
> in the virtualenv itself - is "the obvious approach". From what I've
> seen of tools like virtualenvwrapper, you're not alone in this. [...] But can
> you acknowledge (and document) that "change your way of working" is
> another alternative to "change Python".
>

Acknowledged. What you say is true - but people wanted to know what the 
benefit would be. I laid out my concrete use-case as a rationale. And as 
you note, I am not alone in this type of development. Sure, I care here 
because it affects my style of development, and there are other styles 
that have other benefits (and tradeoffs). I don't see that this part of 
the proposal would negatively affect those styles.

>
>> While a few people have wondered why I would want this -- hopefully answered
>> above -- I have not heard any opposition to this part of the proposal.
> See above. There has been opposition from a number of people. It's
> relatively mild, simply because it's a niche area and doesn't cause
> huge pain, but it's there. And you seem (based on the above analysis)
> to be overstating the benefits, so the summary here is weighted
> heavily in favour of change.

If I have misrepresented anyone, I am sorry - but to the best of my 
understanding, no one had (prior to you, right now) objected to *this 
part* of the proposal. Mark, at least, specified that his concern was 
with the moving of the python binary and that he didn't care about this 
part. I believe Tim indicated that too, but perhaps I have on my 
rose-colored glasses and misunderstood him.

> Also, you have made no attempt that I've seen to address the question
> of why this is important enough to break backward compatibility. Maybe
> it is - but why? Backward compatibility is a very strong rule, and
> should be broken only with good justification. Consistency, and "it
> makes my way of working easier" really shouldn't be sufficient.

In general, yes, I agree with you. However, the break with backwards 
compatibility is, as you point out, minor, and there is a benefit to 
consistency - especially given virtualenv-centric development.


> Has anyone checked whether this will affect people like Enthought and
> ActiveState who distribute their own versions of Python? Is
> ActiveState's PPM tool affected?

I have been running like this for several years across multiple Python 
versions, so I have experience with the "breakage" from this part of the 
proposal. I have found four packages that would need to be updated: Pip, 
virtualenv, PyPM, and Egginst would need 1-2 line patches. I have these 
patches, I would/could provide them. Generally these tools have 
something like:

if platform == 'win32':
   bin_dir = 'Scripts'
else:
   bin_dir = 'bin'

The patches just remove the special casing - bin_dir just gets set to 'bin'.

> Note - that is not "Regularizing the layout". You have not made any
> changes to OS/2 (which matches Windows at the moment). And it doesn't
> match Posix at all.

See my 'summarizing' content above - layouts should match. I also didn't 
want to post chunks of sysconfig.py. You also missed 
distutils.command.install, which is subtly different.

The OS X framework is already posixy inside, and virtualenvs on Mac OS X 
follow the posix-user layout. It is true that some Linux distributions 
place lib, include, etc in the system-wide directories. However, 
altinstalls that are confined to a directory (i.e., have a 'layout' in 
the sense that I am describing.

My proposal would prohibit people from having multiple versions 
installed on top of each other, but they would clobber each other anyway 
with the current setup.


>
>> Again, I have not heard *anyone* objecting to this part of the proposal as
>> it is laid out here. (Paul had a concern with the lib directory earlier, but
>> he said he was ok with the above).
> That's somewhat odd, as I did hear a number of concerns. But it was
> certainly not easy to tell which related to which part of the
> proposal. And all of the objections were mild, mostly because it's not
> a huge practical issue either way.

True. And again, I tried not to misrepresent anyone, but I have not 
heard anyone (including you) who would actually have code broken by 
*this* change. Do you have code? I really, really would like to know. I 
have not found anything that needs a patch except for the four packages 
above.

> Personally, my main concerns are around procedure and policy. The more 
> the discussion goes on, the more I feel that there should be a PEP to 
> capture the details of the debate clearly. Too much is getting lost in 
> the noise. And I think you should provide a clear statement of why 
> this issue is important enough to justify violating the backward 
> compatibility policies. As Mark said (I think it was Mark...) if this 
> had been proposed for 3.0, it would have been fine. Now we're at 3.2 
> with 3.3 close to release, and it just seems too late to be worth the 
> risk. One plus point about your posting this separately. It's made me 
> think through the issue in a bit more detail, and I'm now a solid -1 
> on the proposal. 

I have been trying at various PyCons and in various conversations to 
move this for years. No one cares. The current urgency is driven by 
pyvenv - changes now will be much, much easier than changes later.

Again, I am happy to write a PEP. If I were to summarize (on this issue 
only):

1. The current backwards compatibility hit is minimal; I would be happy 
to contact and provide patches to the four packages I have found (and 
anyone else who wants one). Backwards compatibility in the future will 
probably be harder to deal with.
2. There are advantages to cross-platform consistency and to 
virtualenv-based development. I believe that these will grow in the future.
3. Most people won't care. To the extent that people notice, I think 
they will appreciate the consistency.