[Numpy-discussion] Which Python to use for Mac binaries

[Ralf Gommers]

On Tue, Jan 8, 2013 at 3:12 AM, Ondřej Čertík <ondrej.certik at gmail.com>wrote:

> On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >
> >
> >
> > On Sun, Jan 6, 2013 at 3:21 AM, Ondřej Čertík <ondrej.certik at gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> Currently the NumPy binaries are built using the pavement.py script,
> >> which uses the following Pythons:
> >>
> >> MPKG_PYTHON = {
> >>         "2.5":
> >> ["/Library/Frameworks/Python.framework/Versions/2.5/bin/python"],
> >>         "2.6":
> >> ["/Library/Frameworks/Python.framework/Versions/2.6/bin/python"],
> >>         "2.7":
> >> ["/Library/Frameworks/Python.framework/Versions/2.7/bin/python"],
> >>         "3.1":
> >> ["/Library/Frameworks/Python.framework/Versions/3.1/bin/python3"],
> >>         "3.2":
> >> ["/Library/Frameworks/Python.framework/Versions/3.2/bin/python3"],
> >>         "3.3":
> >> ["/Library/Frameworks/Python.framework/Versions/3.3/bin/python3"],
> >> }
> >>
> >> So for example I can easily create the 2.6 binary if that Python is
> >> pre-installed on the Mac box that I am using.
> >> On one of the Mac boxes that I am using, the 2.7 is missing, so are
> >> 3.1, 3.2 and 3.3. So I was thinking
> >> of updating my Fabric fab file to automatically install all Pythons
> >> from source and build against that, just like I do for Wine.
> >>
> >> Which exact Python do we need to use on Mac? Do we need to use the
> >> binary installer from python.org?
> >
> >
> > Yes, the one from python.org.
> >
> >>
> >> Or can I install it from source? Finally, for which Python versions
> >> should we provide binary installers for Mac?
> >> For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for
> >> OS X 10.3. There is only 2.7 version for OS X 10.6.
> >
> >
> > The provided installers and naming scheme should match what's done for
> > Python itself on python.org.
> >
> > The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5.
> > This is kind of hard to come by these days, but Vincent Davis maintains a
> > build machine for numpy and scipy. That's already set up correctly, so
> all
> > you have to do is connect to it via ssh, check out v.17.0 in
> ~/Code/numpy,
> > check in release.sh that the section for OS X 10.6 is disabled and for
> 10.5
> > enabled and run it.
> >
> > OS X 10.6 broke support for previous versions in some subtle ways, so
> even
> > when using the 10.4 SDK numpy compiled on 10.6 won't run on 10.5. As
> long as
> > we're supporting 10.5 you therefore need to compile on it.
> >
> > The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I
> > have a 10.6 machine, so I can compile those binaries if needed.
> >
> >>
> >> Also, what is the meaning of the following piece of code in pavement.py:
> >>
> >> def _build_mpkg(pyver):
> >>     # account for differences between Python 2.7.1 versions from
> >> python.org
> >>     if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == "10.6":
> >>         ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch
> >> x86_64 -Wl,-search_paths_first"
> >>     else:
> >>         ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch
> >> ppc -Wl,-search_paths_first"
> >>     ldflags += " -L%s" % os.path.join(os.path.dirname(__file__),
> "build")
> >
> >
> > The 10.6 binaries support only Intel Macs, both 32-bit and 64-bit. The
> 10.3
> > binaries support PPC Macs and 32-bit Intel. That's what the above does.
> Note
> > that we simply follow the choice made by the Python release managers
> here.
> >
> >>
> >>     if pyver == "2.5":
> >>         sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" %
> >> (ldflags, " ".join(MPKG_PYTHON[pyver])))
> >>     else:
> >>         sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, "
> >> ".join(MPKG_PYTHON[pyver])))
> >
> >
> > This is necessary because in Python 2.5, distutils asks for "gcc"
> instead of
> > "gcc-4.0", so you may get the wrong one without CC=gcc-4.0. From Python
> 2.6
> > on this was fixed.
> >
> >>
> >> In particular, the last line gets executed and it then fails with:
> >>
> >> paver dmg -p 2.6
> >> ---> pavement.dmg
> >> ---> pavement.clean
> >> LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc
> >> -Wl,-search_paths_first -Lbuild'
> >> /Library/Frameworks/Python.framework/Versions/2.6/bin/python
> >> setupegg.py bdist_mpkg
> >> Traceback (most recent call last):
> >>   File "setupegg.py", line 17, in <module>
> >>     from setuptools import setup
> >> ImportError: No module named setuptools
> >>
> >>
> >> The reason is (I think) that if the Python binary is called explicitly
> >> with /Library/Frameworks/Python.framework/Versions/2.6/bin/python,
> >> then the paths are not setup properly in virtualenv, and thus
> >> setuptools (which is only installed in virtualenv, but not in system
> >> Python) fails to import. The solution is to simply apply this patch:
> >
> >
> > Avoid using system Python for anything. The first thing to do on any new
> OS
> > X system is install Python some other way, preferably from python.org.
> >
> >>
> >> diff --git a/pavement.py b/pavement.py
> >> index e693016..0c637f8 100644
> >> --- a/pavement.py
> >> +++ b/pavement.py
> >> @@ -449,7 +449,7 @@ def _build_mpkg(pyver):
> >>      if pyver == "2.5":
> >>          sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" %
> >> (ldflags, " ".join(MPKG_PYTHON[pyver])))
> >>      else:
> >> -        sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, "
> >> ".join(MPKG_PYTHON[pyver])))
> >> +        sh("python setupegg.py bdist_mpkg")
> >
> >
> > This doesn't work unless using virtualenvs, you're just throwing away the
> > version selection here. If you can support virtualenvs in addition to
> > python.org pythons, that would be useful. But being able to build
> binaries
> > when needed simply by "paver dmg -p 2.x" is quite useful.
>
>
> Absolutely. I was following the release.sh in the numpy git
> repository, which contains:
>
> paver bootstrap
> source bootstrap/bin/activate
> python setupsconsegg.py install
> paver pdf
> paver dmg -p 2.7
>
> So it is using the virtualenv and it works on Vincent's computer, but
> it doesn't work on my
> other computer.
>

Note that it's only using a virtualenv for this one step (building the
docs). This is because building the docs requires installing numpy first to
be able to extract the docstrings.


> I wanted to make the steps somehow reproducible. I started adding the
> commands needed to setup the Mac (any Mac)
> into my Fabfile here:
>
> https://github.com/certik/numpy-vendor/blob/master/fabfile.py#L98
>
> but I run into the issues above.
>
> Of course, I'll try to just use Vincent's computer, but I would feel
> much better if the numpy release process for Mac didn't depend on one
> particular computer, but rather could be quite easily reproduced on
> any Mac OS X of the right version.
>

It doesn't depend on that one computer of course, it takes only a few
minutes to set up a new Mac. But yes, currently it does require admin
rights to install a framework Python.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130108/33250928/attachment.html>