From a.h.jaffe at gmail.com Thu Oct 6 05:56:04 2016 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu, 6 Oct 2016 10:56:04 +0100 Subject: [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra) In-Reply-To: <969CB433-74EF-4417-93F3-46470E4E21A2@twistedmatrix.com> References:

<06E9F64E-FBA4-4909-B1F0-EC58A4A73E7A@twistedmatrix.com> <74585C70-1FC2-4B9C-8A50-DFD12D4275F0@cwi.nl> <3C7A8CDB-388E-4938-A0A6-A6B8BAB6C4AF@gmail.com>

<9B083D20-E19E-486F-A6E8-AD1167CF8E4B@twistedmatrix.com> <50A5DEF7-0281-4849-B0DA-068357F1DA9D@gmail.com> <162AD212-E975-470F-879C-9BC0E2743F86@twistedmatrix.com> <969CB433-74EF-4417-93F3-46470E4E21A2@twistedmatrix.com> <0184dc3a-5bd8-cf0e-fa59-7b60265755eb@gmail.com> Message-ID: <87824946-b86c-8e92-64db-086893ec0443@gmail.com> On 06/10/2016 20:26, Glyph Lefkowitz wrote: >> On Oct 6, 2016, at 2:56 AM, Andrew Jaffe wrote: >> On 17/09/2016 18:59, Glyph Lefkowitz wrote: >>>> On Sep 17, 2016, at 9:27 AM, Ned Deily wrote: >>>> On 2016-09-13 19:33, Glyph Lefkowitz wrote: >>>>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe wrote: >>>>>> >>>>>> $ ls -lt /Library/Python/2.7/site-packages/ >>>>>> total 0 >>>>>> -rwxr-xr-x 1 root wheel 157 31 Jul 02:36 Extras.pth* >>>>>> -rw-r--r-- 1 root wheel 119 31 Jul 02:36 README >>>>>> $ more /Library/Python/2.7/site-packages/Extras.pth >>>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python >>>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC >>>>>> >>>>> Hah! Thanks for sharing. Very satisfying to actually make a *correct* >>>>> prediction about setuptools' behavior :) >>>> >>>> This seems to be Apple's doing. AFAICT, 10.12 is shipping with this >>>> Extras.pth file in /Library/Python/2.7; it's something new. And, >>>> unfortunately, due to https://bugs.python.org/issue4865, the >>>> site-packages directory for the system Python 2.7 is included in >>>> sys.path along with the non-system framework Python site-packages. >>> >> So, a little more data: >> >> If you rename or remove /Library/Python/2.7/site-packages/Extras.pth >> then pip2 works. > > What do you mean by "works"? Your original error is pip refusing to > upgrade pyObjC because to upgrade pyObjC it has to delete the old > version, and pyObjC is part of the operating system, and it can't delete > the installed version. This is correct; the error reporting could be > nicer, but pip is not broken. Don't mess with files in /System. > > The suggestion to use virtualenv isn't really optional. If you really, > really want things to be importable by a bare 'python', not inside a > venv, `pip install --user` is another option you might have. > >> *However*, lots of other stuff breaks -- anything that uses Apple's >> python and relies on access to pyobjc and the frameworks (e.g., >> TextMate's latex package). > > Yep, that's expected behavior. This is exactly why Apple is making it > increasingly difficult to screw up /System. Apps can, should, and do > rely upon the libraries installed on the system. > >> What I don't understand is: what changed from Yosemite? This file did >> not exist before Sierra, but there were no problems with (Apple) >> python accessing these packages. > > Do you mean from El Capitan? Yes, sorry. > This file previously existed in a different location, and (while the > particulars stubbornly refuse to stick to memory for me, for some > reason) this new location is the one generally preferred by the python > packaging maintainers. > >> (Or is there something unique in my setup that is causing this? I kind >> of doubt it, but it's possible...) >> >> Does anyone have any insight? > > If you really, really, really want to do this in such a way that /System > stuff is only present for /System's python and not for python.org > 's, you can take advantage of the 'import' hack > , and > rewrite /Library/Python/2.7/site-packages/Extras.pth to _conditionally_ > add those entries to sys.path, checking sys.executable or some other > fingerprint. > > But: don't. Given that system python and python.org > python share /Library and ~/Library, they're not /really/ distinct > environments, and things installed into one will show up in the other, > so excluding the /System directories on one but not the other will just > cause other, more confusing issues. > > In conclusion: just use virtualenv. This is not a problem that should > be fixed. I agree that this would solve all of the problems, at the cost of some minor startup pain. But one nice thing about the python.org builds is that, for the last few releases, they just worked out of the box, where by "worked" I mean that (as far as I can tell) the system Python saw its own packages, and the python.org build saw its own packages, and (python.org) pip didn't seem to care about the system files. And I suppose I still don't understand exactly why that changed with Sierra, and whether the change is actually beneficial in any way whatever! (And whether it could only be fixed by Apple, or whether a change in the python.org build could do something about it.) Sorry to keep harping on about it, but I find it hard to believe this is not a fairly widespread problem! (Well, I found at least one other complaint: https://jasonkratz.com/2016/09/25/python-2-7-on-the-mac-site-package-weirdness/) Andrew From ronaldoussoren at mac.com Tue Oct 11 17:26:13 2016 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 11 Oct 2016 23:26:13 +0200 Subject: [Pythonmac-SIG] [Pyobjc-dev] PyObjC and macOS 10.12 (Sierra) In-Reply-To: <969CB433-74EF-4417-93F3-46470E4E21A2@twistedmatrix.com> References:

<06E9F64E-FBA4-4909-B1F0-EC58A4A73E7A@twistedmatrix.com> <74585C70-1FC2-4B9C-8A50-DFD12D4275F0@cwi.nl> <3C7A8CDB-388E-4938-A0A6-A6B8BAB6C4AF@gmail.com>

<9B083D20-E19E-486F-A6E8-AD1167CF8E4B@twistedmatrix.com> <50A5DEF7-0281-4849-B0DA-068357F1DA9D@gmail.com> <162AD212-E975-470F-879C-9BC0E2743F86@twistedmatrix.com> <969CB433-74EF-4417-93F3-46470E4E21A2@twistedmatrix.com> Message-ID: > On 17 Sep 2016, at 19:59, Glyph Lefkowitz wrote: > >> >> On Sep 17, 2016, at 9:27 AM, Ned Deily wrote: >> >> On 2016-09-13 19:33, Glyph Lefkowitz wrote: >>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe >>> > wrote: >>>> >>>> Aha! >>>> >>>> $ ls -lt /Library/Python/2.7/site-packages/ >>>> total 0 >>>> -rwxr-xr-x 1 root wheel 157 31 Jul 02:36 Extras.pth* >>>> -rw-r--r-- 1 root wheel 119 31 Jul 02:36 README >>>> $ more /Library/Python/2.7/site-packages/Extras.pth >>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python >>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC >>>> >>>> Now I wonder how those got there?! >>>> >>> >>> Hah! Thanks for sharing. Very satisfying to actually make a *correct* >>> prediction about setuptools' behavior :) >> >> This seems to be Apple's doing. AFAICT, 10.12 is shipping with this >> Extras.pth file in /Library/Python/2.7; it's something new. And, >> unfortunately, due to https://bugs.python.org/issue4865, the >> site-packages directory for the system Python 2.7 is included in >> sys.path along with the non-system framework Python site-packages. > > Hrm. I guess everyone I knew on the beta was using homebrew python :(. > > I'm surprised that Apple is putting stuff in /Library. I don't have a Sierra box handy - /Library isn't SIP-protected now, is it? > > This seems wrong; someone should file a radar (and probably share on http://www.openradar.me for further discussion). Did someone file a radar for this? Otherwise I can do so. I can affirm that Extras.pth is now in /Library/Python/2.7 instead of in a system location. This sucks as bits of the system install now infect other installs of Python 2.7. One more reason to stop using 2.7 I guess ;-) BTW. Has anyone experience with using LLDB on Sierra? I?m currently running Sierra in a VM and for some reason LLDB doesn?t appear to work, in an SSH session I cannot start programs at all and on the console a crashing bug in the interpreter exists the interpreter instead of breaking into the debugger. That?s rather annoying when you?re trying to a debug a crash on 10.12 that doesn?t happen on 10.11. Ronald > > -glyph > > > ------------------------------------------------------------------------------ > _______________________________________________ > Pyobjc-dev mailing list > Pyobjc-dev at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pyobjc-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.h.jaffe at gmail.com Wed Oct 12 14:49:41 2016 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Wed, 12 Oct 2016 19:49:41 +0100 Subject: [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra) In-Reply-To: References:

<06E9F64E-FBA4-4909-B1F0-EC58A4A73E7A@twistedmatrix.com> <74585C70-1FC2-4B9C-8A50-DFD12D4275F0@cwi.nl> <3C7A8CDB-388E-4938-A0A6-A6B8BAB6C4AF@gmail.com>

<9B083D20-E19E-486F-A6E8-AD1167CF8E4B@twistedmatrix.com> <50A5DEF7-0281-4849-B0DA-068357F1DA9D@gmail.com> <162AD212-E975-470F-879C-9BC0E2743F86@twistedmatrix.com> <969CB433-74EF-4417-93F3-46470E4E21A2@twistedmatrix.com> <0184dc3a-5bd8-cf0e-fa59-7b60265755eb@gmail.com> <87824946-b86c-8e92-64db-086893ec0443@gmail.com> Message-ID: > On Oct 7, 2016, at 1:36 AM, Andrew Jaffe wrote: > > On 06/10/2016 20:26, Glyph Lefkowitz wrote: >>> On Oct 6, 2016, at 2:56 AM, Andrew Jaffe wrote: >>> On 17/09/2016 18:59, Glyph Lefkowitz wrote: >>>>> On Sep 17, 2016, at 9:27 AM, Ned Deily wrote: >>>>> On 2016-09-13 19:33, Glyph Lefkowitz wrote: >>>>>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe wrote: >>>>>>> >>>>>>> $ ls -lt /Library/Python/2.7/site-packages/ >>>>>>> total 0 >>>>>>> -rwxr-xr-x 1 root wheel 157 31 Jul 02:36 Extras.pth* >>>>>>> -rw-r--r-- 1 root wheel 119 31 Jul 02:36 README >>>>>>> $ more /Library/Python/2.7/site-packages/Extras.pth >>>>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python >>>>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC >>>>>>> >>>>>> Hah! Thanks for sharing. Very satisfying to actually make a *correct* >>>>>> prediction about setuptools' behavior :) >>>>> >>>>> This seems to be Apple's doing. AFAICT, 10.12 is shipping with this >>>>> Extras.pth file in /Library/Python/2.7; it's something new. And, >>>>> unfortunately, due to https://bugs.python.org/issue4865, the >>>>> site-packages directory for the system Python 2.7 is included in >>>>> sys.path along with the non-system framework Python site-packages. >>>> >>> So, a little more data: >>> >>> If you rename or remove /Library/Python/2.7/site-packages/Extras.pth >>> then pip2 works. >> >> What do you mean by "works"? Your original error is pip refusing to >> upgrade pyObjC because to upgrade pyObjC it has to delete the old >> version, and pyObjC is part of the operating system, and it can't delete >> the installed version. This is correct; the error reporting could be >> nicer, but pip is not broken. Don't mess with files in /System. >> >> The suggestion to use virtualenv isn't really optional. If you really, >> really want things to be importable by a bare 'python', not inside a >> venv, `pip install --user` is another option you might have. >> >>> *However*, lots of other stuff breaks -- anything that uses Apple's >>> python and relies on access to pyobjc and the frameworks (e.g., >>> TextMate's latex package). >> >> Yep, that's expected behavior. This is exactly why Apple is making it >> increasingly difficult to screw up /System. Apps can, should, and do >> rely upon the libraries installed on the system. >> >>> What I don't understand is: what changed from Yosemite? This file did >>> not exist before Sierra, but there were no problems with (Apple) >>> python accessing these packages. >> >> Do you mean from El Capitan? > > Yes, sorry. > >> This file previously existed in a different location, and (while the >> particulars stubbornly refuse to stick to memory for me, for some >> reason) this new location is the one generally preferred by the python >> packaging maintainers. >> >>> (Or is there something unique in my setup that is causing this? I kind >>> of doubt it, but it's possible...) >>> >>> Does anyone have any insight? >> >> If you really, really, really want to do this in such a way that /System >> stuff is only present for /System's python and not for python.org >> 's, you can take advantage of the 'import' hack >> , and >> rewrite /Library/Python/2.7/site-packages/Extras.pth to _conditionally_ >> add those entries to sys.path, checking sys.executable or some other >> fingerprint. >> >> But: don't. Given that system python and python.org >> python share /Library and ~/Library, they're not /really/ distinct >> environments, and things installed into one will show up in the other, >> so excluding the /System directories on one but not the other will just >> cause other, more confusing issues. >> >> In conclusion: just use virtualenv. This is not a problem that should >> be fixed. > > I agree that this would solve all of the problems, at the cost of some minor startup pain. > > But one nice thing about the python.org builds is that, for the last few releases, they just worked out of the box, where by "worked" I mean that (as far as I can tell) the system Python saw its own packages, and the python.org build saw its own packages, and (python.org) pip didn't seem to care about the system files. Trust me, they didn't just work :). There were numerous potential misconfigurations that dealt with conflicts between system python packages and user-installed packages. Installing stuff into the global environment has always been a bad idea. Some system files would be on the path, some wouldn't, and things installed into one environment would bleed over into the other. > And I suppose I still don't understand exactly why that changed with Sierra, and whether the change is actually beneficial in any way whatever! (And whether it could only be fixed by Apple, or whether a change in the python.org build could do something about it.) I'm not sure as to the exact difference, but my basic understanding is that rather than failing silently and ignoring system-installed stuff, pip now reliably and correctly fails to upgrade system packages that are protected by SIP. > Sorry to keep harping on about it, but I find it hard to believe this is not a fairly widespread problem! (Well, I found at least one other complaint: https://jasonkratz.com/2016/09/25/python-2-7-on-the-mac-site-package-weirdness/) The problem is that we're not properly getting the message out about how to set up Python for development; venvs are increasingly graduating from "best practice" to "necessary to function", but I don't think users are getting the message. (Case in point: this thread is still going :).) System packages are for system developers. If you aren't shipping Python.org python itself, or working for apple on OS X, you should not be installing things into /Library or even as a user with *permission* to install things into /Library. -glyph From glyph at twistedmatrix.com Wed Oct 12 02:08:23 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 12 Oct 2016 06:08:23 -0000 Subject: [Pythonmac-SIG] [Pyobjc-dev] PyObjC and macOS 10.12 (Sierra) In-Reply-To: References:

<06E9F64E-FBA4-4909-B1F0-EC58A4A73E7A@twistedmatrix.com> <74585C70-1FC2-4B9C-8A50-DFD12D4275F0@cwi.nl> <3C7A8CDB-388E-4938-A0A6-A6B8BAB6C4AF@gmail.com>

<9B083D20-E19E-486F-A6E8-AD1167CF8E4B@twistedmatrix.com> <50A5DEF7-0281-4849-B0DA-068357F1DA9D@gmail.com> <162AD212-E975-470F-879C-9BC0E2743F86@twistedmatrix.com> <969CB433-74EF-4417-93F3-46470E4E21A2@twistedmatrix.com> Message-ID: <3C564716-E0CE-49DF-BC8C-CE74C568E8CE@twistedmatrix.com> > On Oct 11, 2016, at 2:26 PM, Ronald Oussoren wrote: > > BTW. Has anyone experience with using LLDB on Sierra? I?m currently running Sierra in a VM and for some reason LLDB doesn?t appear to work, in an SSH session I cannot start programs at all and on the console a crashing bug in the interpreter exists the interpreter instead of breaking into the debugger. That?s rather annoying when you?re trying to a debug a crash on 10.12 that doesn?t happen on 10.11. Ever since the introduction of SIP, system-level debugging tools like LLDB have worked very poorly for me, even on self-built executables. I don't see a difference in its behavior on 10.11 vs. 10.12 though. I can run it on `curl` from Homebrew, but not `python`; my guess being that Python is trying to dlopen() some SIP-protected thing whereas `curl` is loading only things from Homebrew? If it's a VM, then system integrity is less of a concern; have you tried just blanket disabling SIP to see if that improves the situation? I've had some luck with other tools (dtrace, mostly) working in that configuration. (I had to totally disable SIP though, disabling one flag at a time didn't work.) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Oct 12 15:09:50 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 12 Oct 2016 19:09:50 -0000 Subject: [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra) In-Reply-To: <63fdcc3f-893f-39b8-b766-f48ae0633e7d@gmail.com> References:

<06E9F64E-FBA4-4909-B1F0-EC58A4A73E7A@twistedmatrix.com> <74585C70-1FC2-4B9C-8A50-DFD12D4275F0@cwi.nl> <3C7A8CDB-388E-4938-A0A6-A6B8BAB6C4AF@gmail.com>