[IPython-dev] [IPython-user] A bag of pythons on OS X leopard; Is a house cleaning in order.

[Barry Wark]

On Wed, Oct 8, 2008 at 10:23 AM, Robin <robince at gmail.com> wrote:
> On Wed, Oct 8, 2008 at 5:55 PM, Barry Wark <barrywark at gmail.com> wrote:
>> On Wed, Oct 8, 2008 at 1:01 AM, Robin <robince at gmail.com> wrote:.
>>
>> Not true. The system python, by default, will put its version of numpy
>> first on the python path (to protect Apple-installed tools that depend
>> on numpy). The system python comes with setuptools, however, for a
>> reason. If you use setuptools to install a more recent version of
>> numpy, setuptools will take care of putting the new version of numpy
>> first on the python path while allowing Apple-installed tools to
>> continue to use the version that shipped with Leopard.
>>
>> So the general rule is: use setuptools and things will "just work".
>
> That sounds great - but I'm curious as to how it can work without
> modifying stuff...
>
> If I have a raw system python prompt and I do "import numpy" - what
> determines which one I get? Sometimes I might want the new numpy (ie
> when working on my own scripts) but sometimes I (or some downloaded
> application assuming a standard system Python install) might want the
> old numpy. If both of these are just doing "import numpy" how does
> setuptools decide which one to serve up?

Good question. By default the system python 2.5 path looks (in
schematic form) like:
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/*:/Library/Python/2.5/lib/site-packages/*

The system python is configured so that distultils installs into
/Library/Python/2.5/site-packages. So, anything installed by distutils
will end up _after_ the system-installed python. So if you install a
new numpy using distutils, and then 'import numpy', you'll get the
system-installed (old) version. This caused quite a bit of confusion
by Leopard python users early on (but see below). System tools can do
the conservative thing and set their python path to ONLY the
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/*
tree and are thus immune to changes or new packages in
/Library/Python.

OK, but what about setuptools? Packages installed using setuptools (or
the easy_install frontend) go into /Library/Python as well, but
setuptools can, at runtime, move those setuptools-installed packages
to the front of the python path so that your new, setuptools-installed
numpy is what you'll get when you do 'import numpy'. Conservative
packages or scripts (e.g. system tools) can (and do) request numpy by
version from setuptools (it turns out even the
/System/.../site-package directory uses setuptools) the version that
they're tested against using. Thus, you get the latest version via
'import numpy' but can request an older numpy by version explicitly.
Make sense? It's a bit tricky and it took me quite a while to groking
the whole setup enough to trust it, but in general it does the "right
thing" (how very Apple). It's possible that there are some

The only downside to this system, that I'm aware of, is for users
whose /Library folder is mounted via NFS. Setuptools apparently
doesn't play very well with NFS (because it generates many more file
reads than without setuptools) so imports can take longer when using
setuptools than if you go with distutils only. In practice, I don't
find that this is a problem, but folks that are using ipython on a
large cluster have found it onerous. If this is the case for you, then
Fink/MacPorts or MacPython is probably your only option at this point.

Finally, there are a few (long) threads on this topic on both the
numpy-discussion and pythonmac-sig lists including comments from folks
much more knowlegeable than I. You may find more insight by searching
those archives.

>
> Another possible reason for using seperate python - if an operating
> system upgrade changes the python version (ie 2.5 -> 2.6) then won't
> that break all your installed modules built against the old version?

No, the python /Library/Python/2.X directories are segregated by
python version (note that there're both python 2.3 and 2.5 installed
by default on Leopard). So when Apple releases a system python 2.6 or
some other python version, your existing installed packages will not
break, but they will not be available to the new version (remember,
separate site-packages for each version). You will have to
rebuild/install packages for the new python version.

>
> If using macports python, there is a good chance it'll carry on
> working after an upgrade... (it did for me with Leopard)

True.

>
> Robin
>