[Python-Dev] PEP 370, open questions

[Ronald Oussoren]

On 17 Jan, 2008, at 9:40, glyph at divmod.com wrote:

>
> On 07:55 am, lists at cheimes.de wrote:
>>
>
>> The PEP 370 (http://www.python.org/dev/peps/pep-0370) "per user site
>> packages directory" has several open questions:
>>
>> * Are the directories for Windows, Mac and Unix fine?
>> * Mac: Should framework and non-framework builds of Python use
>> the same directories?
>
> One of the problems we've encountered over and over again with
> Combinator is that MacOS has a dual personality.  Its personality is  
> not
> just an issue of framework vs. non-framework build, but a fundamental
> question of "is this platform a UNIX or is it not a UNIX".

IMHO the question is wrong. MacOS X is a UNIX platform (and Apple has  
the paperwork to prove it), but at the same time it has a non- 
traditional filesystem layout.

The "darwin" system at the foundation of OSX is a classic unix system,  
with the usual filesystem layout, but the bits that make OSX OSX use a  
completely diffent filesystem layout.

The framework build of Python definitly targets the upper layer of the  
OSX stack, not just the Unix core. This sometimes causes confusion,  
but mostly from developers wandering over from a Linux system that  
complain that OSX isn't Linux.

Note that even Darwin is not Linux, there are several differences that  
cause problems for naive porters. Two of those: Darwin uses different  
binary formats for shared libraries and plugins;  the darwin linker  
hardcodes the full path to shared libraries into executables (without  
a runtime search path).

And that's only the start of fun, universal (fat) binaries make live  
even more interesting.

>
>
> An example: Let's say you have a GNU autotools project in C, which  
> we'll
> call "Xxx", and a set of Python bindings, PyXxx.  Combinator deals  
> with
> this by using ~/.local, and providing scripts to set up PATH and
> LD_LIBRARY_PATH to point to ~/.local/bin and ~/.local/lib,  
> respectively.

~/Library/Combinator would be a better installation root on OSX, that  
location fits better into guidelines from Apple and also avoids  
completely hiding the Combinator data from the user.

>
> I'm not suggesting that Python come with such a tool (although it  
> might
> be handy, at least until this convention catches on with  
> distributors),
> but it should at least make it clear how one would achieve the  
> following
> straightforwardly:

This is probably off-topic for python-dev, but how is combinator  
different from zc.buildout and virtualenv?

I'm particularly liking virtualenv at the moment as it allows me to  
have seperate trees for projects without having several full python  
installations.

>
>
>  cd xxx-0.2.4
>  ./configure --prefix ~/.local
>  make
>  make install
>  cd ../pyxxx-0.0.1
>  python setup.py install --prefix ~/.local
>
> Using Combinator, the user is now fully set up to "import xxx".  But
> only if they're using Linux, because I made the same mistake (which we
> will probably be correcting at some point soon) of using Python's
> *existing* per-user site-packages directory of ~/Library/Python on the
> mac, and not adding ~/.local.

Why? Just because users can't remember on which platform they are  
developing ;-)? That said, there's a libpython.a file in the framework  
build for basicly that reason: enough users complained about there not  
being a python library they could link to that it was easier to add a  
symlink then trying to educate everyone.

>
>
> On the Mac, our user now has a problem: given that ~/Library/Python/
> doesn't follow the /usr or /usr/local style filesystem layout,
> autotools-based projects will not build their libraries in the right
> places using a single command-line option.  (I guess maybe you could  
> do
> something with --bindir and --sbindir and --exec-prefix or whatever, I
> don't know.  I tend to get bored and wander off when I have to pass  
> more
> than 3 options to a configure script.)

You could teach Combinator about running configure scripts ;-).


> A user familiar with the Mac alone might know what to do at this point
> (to be honest, I don't!), but I do know that people familiar with both
> platforms are confused by this apparently gratuitous inconsistency.

This is not a gratuitous inconsistency.  The Mac developers (mostly  
Jack Jansen, but I was involved as well) decided that following the  
platform conventions is a good thing.

>
> They follow familiar procedure on a Linux box, it works great, then  
> they
> do the same thing on a Mac (with the same shell, an apparently similar
> version of Python) and it doesn't.  Keep in mind that "./configure
> --prefix" is familiar procedure from a lot more places than  
> Combinator.
> For example, on shared hosting where I didn't have root, I've used  
> this
> same trick without Combinator, building Python itself with --prefix
> ~/.local and editing .bashrc to modify the appropriate env vars.
>
> My suggestion would be to have *both* ~/.local *and* ~/Library/ 
> Python be
> honored on the Mac, because there really isn't much harm in doing so.
> Perhaps it would make sense for non-framework builds to not honor
> ~/Library/Python, but I am pretty certain, based on experience  
> fielding
> requests from confused users, that framework builds should still honor
> ~/.local/lib/python.../site-packages.

As long as the default installation location is still ~/Library/Python/ 
X.Y for framework builds I wouldn't mind too much.


>
> Windows has this problem less because everything has to be done so
> completely differently.
>
> In any event, the really important thing from my perspective is that  
> the
> PEP should explain how this very common use-case for per-user
> installation of libraries can be accomplished on each of the "big  
> three"
> platforms.  This explanation should be put somewhere that users can
> easily find it when they are building libraries.

I don't know if the PEP should mention this, but I do agree that this  
should be in the Python documentation.


Ronald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2224 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20080117/b3bcf94f/attachment-0001.bin