From matthew.brett at gmail.com  Fri Jul  1 05:37:03 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 1 Jul 2016 10:37:03 +0100
Subject: [Pythonmac-SIG] Delete load command (install name) from dylib?
Message-ID: <CAH6Pt5osmymw-h9rSu44y1m=HTnvr4eho0UOtkFP2trffvx+Xg@mail.gmail.com>

Hi,

I'm sorry if this is an easy question to which I have missed an
obvious answer, but

Is there any way (using macholib or otherwise) to delete a library
dependency?  Specifically, let's say I have a library with
dependencies like this:

$ otool -L libdlib.19.0.99.dylib

libdlib.19.0.99.dylib:
/Users/mb312/dev_trees/dlib-wheels/dlib/dlib/libdlib.19.0.99.dylib
(compatibility version 0.0.0, current version 19.0.99)
/opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1197.1.1)

and I want to delete the dependency on /opt/X11/lib/libSM.6.dylib - is
there a way do this?  As in something like:

delete_install_name /opt/X11/lib/libSM.6.dylib libdlib.19.0.99.dylib

Thanks for any insights,

Matthew

From ronaldoussoren at mac.com  Sat Jul  2 01:58:13 2016
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 02 Jul 2016 14:58:13 +0900
Subject: [Pythonmac-SIG] Delete load command (install name) from dylib?
In-Reply-To: <CAH6Pt5osmymw-h9rSu44y1m=HTnvr4eho0UOtkFP2trffvx+Xg@mail.gmail.com>
References: <CAH6Pt5osmymw-h9rSu44y1m=HTnvr4eho0UOtkFP2trffvx+Xg@mail.gmail.com>
Message-ID: <C601AB2D-F084-4082-B432-9D86BDE53411@mac.com>

I don't think this is possible with macholib, py2app is the primary use case for developing macholib and doesn't need this functionality.  

Why do you want to do this? I'd expect there to be a real dependency on the library (by using a symbol from it) and removing the load command would just cause a crash. It might be possible to change the load command to a weak linking one, although I don't know if that would require changes to the loaded symbols as well. Anyway, afaik macholib doesn't support that either (yet). 

Ronald

--
On the road, hence brief. 

Op 1 jul. 2016 om 18:37 heeft Matthew Brett <matthew.brett at gmail.com> het volgende geschreven:

> Hi,
> 
> I'm sorry if this is an easy question to which I have missed an
> obvious answer, but
> 
> Is there any way (using macholib or otherwise) to delete a library
> dependency?  Specifically, let's say I have a library with
> dependencies like this:
> 
> $ otool -L libdlib.19.0.99.dylib
> 
> libdlib.19.0.99.dylib:
> /Users/mb312/dev_trees/dlib-wheels/dlib/dlib/libdlib.19.0.99.dylib
> (compatibility version 0.0.0, current version 19.0.99)
> /opt/X11/lib/libSM.6.dylib (compatibility version 7.0.0, current version 7.1.0)
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1197.1.1)
> 
> and I want to delete the dependency on /opt/X11/lib/libSM.6.dylib - is
> there a way do this?  As in something like:
> 
> delete_install_name /opt/X11/lib/libSM.6.dylib libdlib.19.0.99.dylib
> 
> Thanks for any insights,
> 
> Matthew
> _______________________________________________
> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> https://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG

From matthew.brett at gmail.com  Sat Jul  2 04:18:04 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 2 Jul 2016 09:18:04 +0100
Subject: [Pythonmac-SIG] Delete load command (install name) from dylib?
In-Reply-To: <C601AB2D-F084-4082-B432-9D86BDE53411@mac.com>
References: <CAH6Pt5osmymw-h9rSu44y1m=HTnvr4eho0UOtkFP2trffvx+Xg@mail.gmail.com>
 <C601AB2D-F084-4082-B432-9D86BDE53411@mac.com>
Message-ID: <CAH6Pt5q8EpE8FrJ+hMJnjBJ1CL0QKEABygEDdj5YdebYokSDqA@mail.gmail.com>

Hi,

On Sat, Jul 2, 2016 at 6:58 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> I don't think this is possible with macholib, py2app is the primary use case for developing macholib and doesn't need this functionality.
>
> Why do you want to do this? I'd expect there to be a real dependency on the library (by using a symbol from it) and removing the load command would just cause a crash. It might be possible to change the load command to a weak linking one, although I don't know if that would require changes to the loaded symbols as well. Anyway, afaik macholib doesn't support that either (yet).

Thanks for the reply.

The specific case I'm working on is trying to fix some mis-features of
the way that VTK is built on OSX.   VTK gets built with cmake, and
cmake does not seem to have a good general way of dealing with the
standard method of Python linking, that is, leaving the Python symbols
undefined, to be looked up at run time [1, 2].

For example the VTK binary installer for OSX [3] has its own copy of
Python, and links directly to system Python for its symbols, making it
useless for working with other Pythons or in virtualenvs.

There's some work underway in VTK [2, 4] and cmake [5] to fix cmake to
label the Python symbols as undefined, and to look these symbols up at
run-time, but this is not yet merged with VTK or implemented in cmake.

So, I was trying some surgery on the distributed VTK installer to make
it suitable for a binary wheel, by setting the library links to be
relative to the vtk installation directory, and removing the direct
linking to the system Python binary [6].

The problem is the one you pointed out, which is that the symbol table
of the various libs specifically tell the linker to go get the Python
symbols from the linked Python binary.  For example, before deleting
the Python system library from the list of libraries to be loaded:

    $ nm -m vtk/lib/libvtkWrappingPython27Core-7.0.1.dylib | grep _PyBase
                 (undefined) external _PyBaseObject_Type (from Python)

This should be:

     (undefined) external _PyBaseObject_Type (dynamically looked up)

So, yes, you're right - just deleting the library from the list isn't
enough, it requires rewriting the symbol table.  It's tough because,
if I could do that, it would be possible to generate a functioning and
very useful VTK binary wheel from the current VTK install package.

Cheers,

Matthew

[1] http://blog.tim-smith.us/2015/09/python-extension-modules-os-x/
[2] https://github.com/Homebrew/homebrew-science/issues/3401
[3] http://www.vtk.org/files/release/7.0/vtkpython-7.0.0-Darwin-64bit.dmg
[4] https://gitlab.kitware.com/vtk/vtk/merge_requests/1511
[5] https://docs.google.com/document/d/1e46rbbedffiQLHSRWwjuO9tbscOM9fXhWarL8eNhG5o/edit#
[6] https://gist.github.com/matthew-brett/2907cdb31a042c7a737902b719f4deb8

From ronaldoussoren at mac.com  Sat Jul  2 07:27:59 2016
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 02 Jul 2016 20:27:59 +0900
Subject: [Pythonmac-SIG] Delete load command (install name) from dylib?
In-Reply-To: <CAH6Pt5q8EpE8FrJ+hMJnjBJ1CL0QKEABygEDdj5YdebYokSDqA@mail.gmail.com>
References: <CAH6Pt5osmymw-h9rSu44y1m=HTnvr4eho0UOtkFP2trffvx+Xg@mail.gmail.com>
 <C601AB2D-F084-4082-B432-9D86BDE53411@mac.com>
 <CAH6Pt5q8EpE8FrJ+hMJnjBJ1CL0QKEABygEDdj5YdebYokSDqA@mail.gmail.com>
Message-ID: <CBE0C55C-3B89-44DC-A75F-E2E4115D234E@mac.com>

Fixing the VTK build would be the best way to fix this issue, but from my rather hazy memory building it might not be easy.  If they'd use distutils for building life would be much easier here, and IIRC python-config also returns the right linker flags. 

I don't have time to look into modifying macholib for this, maybe later this year. 

Ronald
--
On the road, hence brief. 

Op 2 jul. 2016 om 17:18 heeft Matthew Brett <matthew.brett at gmail.com> het volgende geschreven:

> Hi,
> 
>> On Sat, Jul 2, 2016 at 6:58 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>> I don't think this is possible with macholib, py2app is the primary use case for developing macholib and doesn't need this functionality.
>> 
>> Why do you want to do this? I'd expect there to be a real dependency on the library (by using a symbol from it) and removing the load command would just cause a crash. It might be possible to change the load command to a weak linking one, although I don't know if that would require changes to the loaded symbols as well. Anyway, afaik macholib doesn't support that either (yet).
> 
> Thanks for the reply.
> 
> The specific case I'm working on is trying to fix some mis-features of
> the way that VTK is built on OSX.   VTK gets built with cmake, and
> cmake does not seem to have a good general way of dealing with the
> standard method of Python linking, that is, leaving the Python symbols
> undefined, to be looked up at run time [1, 2].
> 
> For example the VTK binary installer for OSX [3] has its own copy of
> Python, and links directly to system Python for its symbols, making it
> useless for working with other Pythons or in virtualenvs.
> 
> There's some work underway in VTK [2, 4] and cmake [5] to fix cmake to
> label the Python symbols as undefined, and to look these symbols up at
> run-time, but this is not yet merged with VTK or implemented in cmake.
> 
> So, I was trying some surgery on the distributed VTK installer to make
> it suitable for a binary wheel, by setting the library links to be
> relative to the vtk installation directory, and removing the direct
> linking to the system Python binary [6].
> 
> The problem is the one you pointed out, which is that the symbol table
> of the various libs specifically tell the linker to go get the Python
> symbols from the linked Python binary.  For example, before deleting
> the Python system library from the list of libraries to be loaded:
> 
>    $ nm -m vtk/lib/libvtkWrappingPython27Core-7.0.1.dylib | grep _PyBase
>                 (undefined) external _PyBaseObject_Type (from Python)
> 
> This should be:
> 
>     (undefined) external _PyBaseObject_Type (dynamically looked up)
> 
> So, yes, you're right - just deleting the library from the list isn't
> enough, it requires rewriting the symbol table.  It's tough because,
> if I could do that, it would be possible to generate a functioning and
> very useful VTK binary wheel from the current VTK install package.
> 
> Cheers,
> 
> Matthew
> 
> [1] http://blog.tim-smith.us/2015/09/python-extension-modules-os-x/
> [2] https://github.com/Homebrew/homebrew-science/issues/3401
> [3] http://www.vtk.org/files/release/7.0/vtkpython-7.0.0-Darwin-64bit.dmg
> [4] https://gitlab.kitware.com/vtk/vtk/merge_requests/1511
> [5] https://docs.google.com/document/d/1e46rbbedffiQLHSRWwjuO9tbscOM9fXhWarL8eNhG5o/edit#
> [6] https://gist.github.com/matthew-brett/2907cdb31a042c7a737902b719f4deb8

From matthew.brett at gmail.com  Sat Jul  2 07:58:29 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 2 Jul 2016 12:58:29 +0100
Subject: [Pythonmac-SIG] Delete load command (install name) from dylib?
In-Reply-To: <CBE0C55C-3B89-44DC-A75F-E2E4115D234E@mac.com>
References: <CAH6Pt5osmymw-h9rSu44y1m=HTnvr4eho0UOtkFP2trffvx+Xg@mail.gmail.com>
 <C601AB2D-F084-4082-B432-9D86BDE53411@mac.com>
 <CAH6Pt5q8EpE8FrJ+hMJnjBJ1CL0QKEABygEDdj5YdebYokSDqA@mail.gmail.com>
 <CBE0C55C-3B89-44DC-A75F-E2E4115D234E@mac.com>
Message-ID: <CAH6Pt5qM1pO5r6=AE=YZjN=4ir5C_punrT99zudbAB=WEMEODw@mail.gmail.com>

On Sat, Jul 2, 2016 at 12:27 PM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> Fixing the VTK build would be the best way to fix this issue, but from my rather hazy memory building it might not be easy.  If they'd use distutils for building life would be much easier here, and IIRC python-config also returns the right linker flags.

It's certainly not an easy build, hard enough that I doubt they'd
consider switching to distutils.  Of course they (kitware) also have a
lot of cmake experience.

Cheers,

Matthew

From ronaldoussoren at mac.com  Fri Jul 22 09:46:18 2016
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 22 Jul 2016 15:46:18 +0200
Subject: [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)
Message-ID: <B0429FC2-0608-48BC-9618-1A41CAA8A156@mac.com>

Hi,

I?m slowly working my way through the SDK headers for macOS 10.12 during idle time at EuroPython2016, from the frameworks with small diffs to those with larger diffs.  

There are at the moment about 26 frameworks I haven?t looked at at all, and another 10 where I know I have to do some more work. The rest should have up to date metadata on my machine, but haven?t been tested on macOS 10.12 yet (and that won?t change until I get around to actually installing a VM running 10.12).

All of this was done using the headers in the Xcode 8 beta 2, I haven?t looked at the incremental changes to the SDK in beta 3 yet (and likely won?t until I have made a pass through all frameworks).  That said, the diffs from beta 2 to later beta?s and the final release should be fairly small (he wrote hopefully).

What needs to be done to get proper support for 10.12:

* Finish updating the metadata.  I suspect that this is a couple of days work, I have worked may way through the diffs of a fairly large number of frameworks, but haven?t looked at the more complicated ones yet (such as Foundation and AppKit). Also, a fairly large subset of the frameworks I have looked at didn?t have any significant updates compared to 10.11 (but some of them still had large textual diffs due to restructuring of header files).

* Install a 10.12 VM and run tests there. This will likely result in more work, there?s already a pull request about NSSecureCoder warnings on 10.12 that needs looking into and there?s bound to be more issues.

* Do the build and test dance on older OSX releases. 
   The annoying bit for that: there?s a know issue with building on OSX 10.7, see issue #100. I haven?t been able to look into that yet
   because I don?t have an VM with 10.7 on my laptop and haven?t been able to install 10.7 there yet. I probably have such a VM on
   an external disk somewhere, but I?m not sure about that.

When all that is done I can do a release that includes 10.12 support, with some luck around the time of the 10.12 release itself. I?ll definitely push the metadata updates to bitbucket once I?ve finished looking through the 10.12 SDK diffs.

In the longer run I want to provide wheels for PyObjC, but that requires more work. In particular, I only want to do that after automating more of the release work because I?ve noticed that I?m already postponing doing new releases due to the amount of work and don?t want to make things worse in that regard.

BTW. I?ll likely won?t work on metadata updates during the EP2016 sprints, my current plan is to keep the tradition of working on PEP 447 alive and try to actually get it accepted this year. 

Ronald

PS. ?A couple of days work? probably means that it will take me several calendar weeks to actually finish the work because I have limited time to work on this and don?t want to spent all that time on this rather tedious work.

[#100]: https://bitbucket.org/ronaldoussoren/pyobjc/issues/100/cannot-find-interface-declaration-for