[Pythonmac-SIG] Advice wanted on dependency building...

[Chris Barker - NOAA Federal]

I just poked a bit into the Anaconda Python distribution. their
packages are simple tarballs, but I think they have a dependency
management system of some sort.

They deliver the dependencies as separate packages (tarballs), one for
each lib. It looksl ike it all gets unpacked inot a sinlgle dir (in
/opt), and an example python extension is built like this:

$ otool -L netCDF4.so
netCDF4.so:
	@loader_path/../../libnetcdf.7.dylib (compatibility version 10.0.0,
current version 10.0.0)
	@loader_path/../../libhdf5_hl.7.dylib (compatibility version 8.0.0,
current version 8.3.0)
	@loader_path/../../libhdf5.7.dylib (compatibility version 8.0.0,
current version 8.3.0)
	@loader_path/../../libz.1.dylib (compatibility version 1.0.0, current
version 1.2.7)
	/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 111.0.0)


I don't know how to get that @loader_path thing in there, but this
seems like a reasonable way to do it (though I guess it wouldn't
support virtualenv...)

-Chris



On Wed, May 22, 2013 at 3:46 PM, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
> Thanks Ronald,
>
> On Wed, May 22, 2013 at 2:53 PM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>
>> To move back onto topic, not relying on unix-level libraries in OSX is in a good thing as it makes it easier to support multiple OSX versions with a single set of binaries.
>
> hmm -- I figured if it was a system lib, it should work on whatever
> system It's running on. For example, I'm working right now on the
> netcdf4 lib -- it required hdr5, which requires zlib. I"m using the
> system zlib -- is that a bad idea? Should I build it too, to make sure
> it matches the rest of it?
>
> (I do want the binaries to run anywhere the binary Python I'm using runs)
>
>
>> Except for a number of more complicated libraries (such as PIL/Pillow) when using universal binaries (when using 'pip install', homebrew/macports/... have their own mechanisms for building).
>
> right -- Universal libs are not well supported by those systems -- but
> that's the power users problem!
>
>>> 2) folks that want to use a Mac like a Mac, and people that develop
>>> for those folks --  these people need binary installers, and may want
>>> to be able to use and deploy either packages or applications (Py2app)
>>> that will run on systems older than the one developed on, or want
>>> universal builds, or ???
>>>  - These are the folks I'd like to support, but I'm still unsure as
>>> to how best to do that.
>>
>> It would be nice to have a set of binary "packages", based on a reproducable build system.
>
> Exactly what I'd like to build!
>
>>> Way back when Bob Ippolito maintained a repository of binary packages
>>> for the mac -- it was a great resource, but he's long since moved on
>>> to other things.
>>
>> The binary packages that Bob maintained had IMHO two major problems:
>>
>> 1) The largest problem is that the packages were AFAIK created ad-hoc (Bob or some other contributor did the magic incantations to build library dependencies)
>
> Yeah, and he never gave anyone else permission to push to it...
>
>> 2) The packages were Installer.app packages. The current state of the art for development/project environments is to use virtualenv or buildout to create separated python installations and install all project depedencies there instead of the global site-packages directory. That's not something that's easily supported with Installer.app packages.
>
> It was the way to go at the time, but I agree a binary format that
> supports virtualenv would be great.
>
>>> do I really want linked into my single instance of Python at run time?
>>
>> As long as the libpng state isn't shared static linking isn't really a
>> problem.
>
> good to know, but somehow it still offends my sensibilities
>
>> Dynamic linking has at least two disadvantages:
>>
>> 1) Relocation of the installation prefix is harder due to the way the dynamic linker on OSX looks for libraries:
>
> yeah -- that is a pain.
>
>> The header is easily updated using macholib, but that makes installation
>> harder and isn't supported by the standard packaging tools (easy_install
>> and pip)
>
> But if we put the shared libs in amore central location, then all your
> virtual-ens could use the same ones, yes?
>
>> 2) The primary use case for dynamic linking is to share dylibs between extensions, and when those extensions are in different PyPI packages the packaging story gets more complicated. The easiest workaround is to ignore sharing dylibs and still bundle multipe copies of libpng if two different PyPI packages both link with libpng.
>
> when you say bundle, do you mean static link? Or just package up the
> dylib with the bundle, which is what i was thinking -- each package
> installs the libs it needs, which may or may not already have been
> installed by another package -- but so what?
>
> And I expect the number of folks building packages will be fairly
> small, so one builder would one have to build one set of dylibs.
>
>>> But if dynamic, where do you put them? We'll still want to ship them
>> A new framework isn't necessary. There are three locations that could easily be used:
>>
>> 1) A directory in Python.framework, for example /Library/Frameworks/Python.framework/Frameworks
>
> That makes sense to me.
>
>> 2) A directory in /Library/Python, for example /Library/Python/Externals
>
> that feels a bit lke Apple's turf, but what do I know?
>
>> 3) As 2), but in the users home directory (~/Library/Python/Externals)
>> The latter is the only one where you can install without admin privileges.
>
> But we put the python binaries  in /LIbrary/Frameworks -- it seems we
> should do the same with libs...
>
>
>> The folks over on distutils-sig are working towards support for wheels (PEP 427, <http://www.python.org/dev/peps/pep-0427/>) at least in pip and distribute/setuptools and possibly in the stdlib as well (for 3.4). It would be nice if the OSX package collection would be in wheel format, that would make it relatively easy to install the packages using the defacto standard tools.
>
> Any idea what the time scale is on this?
>
>> What I haven't looked into yet is how easy it would be to configure pip to look for packages on PyPI and then look for binaries (wheels) in some other location.
>
> Have the pip folks made any commitment at all to supporting binary
> installs? That's a big missing feature.
>
>>> Note that I've used the term "we" here ;-)  I'm hoping that others
>>> will join me in following a convention and getting stuff out there,
>>> but even if not, I'd love feedback on how best to do it.
>>
>> Good luck :-).  This is fairly boring low-level packaging work, and that tends to put off people. At least it is a lot easier than trying to fix or replace distutils, that has burned out at least 3 generations of developers ;-/
>
> Well, I'm an optimist -- and recently at least you and Ned and Russel
> Owen have bee known to contribute.
>
>>> By the way, the other goal is to build scripts that do the builds the
>>> way we need for various libs, packages, etc, so that it's easy to do
>>> it all when new builds are required...
>>> (maybe use gattai? -- http://sourceforge.net/projects/gattai/)
>>
>> I haven't used gattai before, but at least it is Python code. Building tools for this from scratch would also not be very hard, I already done so a number of times (such as the build-installer.py script in the CPython repository).
>
> yup -- me too, though I find myself wanting to add various make-like
> features, and it dawns on me that I am re-inventing the wheel! So I
> want to give Gattai a shot. Also Kevin is likely to be helpful.
>
>> The hard part should be setting up the build infrastructure and build scripts, once couple of relatively hard packages like PIL or wx have been processed adding more packages and new versions of packages should be easy.
>
> Exactly. But I don't know about wx -- that is a bear, and Robin's been
> doing a fine job!
>
> Thanks for you ideas,
>
>   -Chris
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov