[Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)

[VanL]

As this has been brought up a couple times in this subthread, I figured 
that I would lay out the rationale here.

There are two proposals on the table: 1) Regularize the install layout, 
and 2) move the python binary to the binaries directory. This email will 
deal with the first, and a second email will deal with the second.

1) Regularizing the install layout:

One of Python's strengths is its cross-platform appeal. Carefully-
written Python programs are frequently portable between operating
systems and Python implementations with very few changes. Over the
years, substantial effort has been put into maintaining platform
parity and providing consistent interfaces to available functionality,
even when different underlying implementations are necessary (such
as with ntpath and posixpath).

One place where Python is unnecessarily different, however, is in
the layout and organization of the Python environment. This is most
visible in the name of the directory for binaries on the Windows 
platform ("Scripts") versus the name of the directory for binaries on 
every other platform ("bin"), but a full listing of the layouts shows
substantial differences in layout and capitalization across platforms.
Sometimes the include is capitalized ("Include"), and sometimes not; and
the python version may or may not be included in the path to the
standard library or not.

This may seem like a harmless inconsistency, and if that were all it 
was, I wouldn't care. (That said, cross-platform consistency is its own 
good). But it becomes a real pain when combined with tools like 
virtualenv or the new pyvenv to create cross-platform development 
environments.

In particular, I regularly do development on both Windows and a Mac, and 
then deploy on Linux. I do this in virtualenvs, so that I have a 
controlled and regular environment. I keep them in sync using source 
control.

The problem comes when I have executable scripts that I want to include 
in my dvcs - I can't have it in the obvious place - the binaries 
directory - because *the name of the directory changes when you move 
between platforms.* More concretely, I can't hg add "Scripts/runner.py? 
on my windows environment (where it is put in the PATH by virtualenv) 
and thendo a pull on Mac or Linux and have it end up properly in 
"bin/runner.py" which is the correct PATH for those platforms.

This applies anytime there are executable scripts that you want to 
manage using source control across platforms. Django projects regularly 
have these, and I suspect we will be seeing more of this with the new 
"project" support in virtualenvwrapper.

While a few people have wondered why I would want this -- hopefully 
answered above -- I have not heard any opposition to this part of the 
proposal.

This first proposal is just to make the names of the directories match 
across platforms. There are six keys defined in the installer files 
(sysconfig.cfg and distutils.command.install): 'stdlib', 'purelib', 
'platlib', 'headers', 'scripts',  and 'data'.

Currently on Windows, there are two different layouts defined:

   'nt': {
     'stdlib': '{base}/Lib',
     'platstdlib': '{base}/Lib',
     'purelib': '{base}/Lib/site-packages',
     'platlib': '{base}/Lib/site-packages',
     'include': '{base}/Include',
     'platinclude': '{base}/Include',
     'scripts': '{base}/Scripts',
     'data'   : '{base}',
     },

   'nt_user': {
     'stdlib': '{userbase}/Python{py_version_nodot}',
     'platstdlib': '{userbase}/Python{py_version_nodot}',
     'purelib': '{userbase}/Python{py_version_nodot}/site-packages',
     'platlib': '{userbase}/Python{py_version_nodot}/site-packages',
     'include': '{userbase}/Python{py_version_nodot}/Include',
     'scripts': '{userbase}/Scripts',
     'data'   : '{userbase}',
     },


The proposal is to make all the layouts change to:

   'nt': {
     'stdlib': '{base}/lib',
     'platstdlib': '{base}/lib',
     'purelib': '{base}/lib/site-packages',
     'platlib': '{base}/lib/site-packages',
     'include': '{base}/include',
     'platinclude': '{base}/include',
     'scripts': '{base}/bin',
     'data'   : '{base}',
     },

The change here is that 'Scripts' will change to 'bin' and the 
capitalization will be removed. Also, "user installs" of Python will 
have the same internal layout as "system installs" of Python. This will 
also, not coincidentally, match the install layout for posix, at least 
with regard to the 'bin', 'lib', and 'include' directories.

Again, I have not heard *anyone* objecting to this part of the proposal 
as it is laid out here. (Paul had a concern with the lib directory 
earlier, but he said he was ok with the above).

Please let me know if you have any problems or concerns with this part 1.

Thanks,
Van