From justvanrossum at gmail.com Mon Oct 1 09:41:44 2018 From: justvanrossum at gmail.com (Just van Rossum) Date: Mon, 1 Oct 2018 15:41:44 +0200 Subject: [Pythonmac-SIG] py2app -A and virtual environments In-Reply-To: References: <10601342-3D10-4A17-965A-82C48C292A31@gmail.com> Message-ID: <7C081100-EA3B-4286-AE36-2AD779307573@gmail.com> > On 28 Sep 2018, at 13:08, Ronald Oussoren wrote: > > > >> On 28 Sep 2018, at 10:34, Just van Rossum wrote: >> >> Following up to this: it seems line 1832 in py2app/build_app.py isn?t correct anymore, with Python 3 venv behavior: >> >> if os.path.exists(os.path.join(sys.prefix, ".Python")): >> >> Not sure how to fix it. If I force this code path to be taken, I get an AttributeError on sys.real_prefix, so I guess a bit more needs to be done. > > The line of code is for virtualenv support. I?ll have to check the code to see what?s needed to support venv here as well. Yes, it?s checking to see whether we?re running from a venv, but it?s not triggering as it should, I think. Somehow this ?.Python? path doesn?t exist in my case, even though I am running from a venv. Just > > Ronald > >> >> Just >> >>> On 27 Sep 2018, at 16:05, Just van Rossum wrote: >>> >>> Hi all, >>> >>> py2app seems to do the right thing from an activated virtual env when building a full app. However, for speedy development I use the py2app -A option a lot, and when running such an alias bundle it appears to use the global Python install instead of the virtual env. Is there a way around that? >>> >>> Just >>> >> > From justvanrossum at gmail.com Mon Oct 1 09:43:09 2018 From: justvanrossum at gmail.com (Just van Rossum) Date: Mon, 1 Oct 2018 15:43:09 +0200 Subject: [Pythonmac-SIG] py2app -A and virtual environments In-Reply-To: References: <10601342-3D10-4A17-965A-82C48C292A31@gmail.com> Message-ID: > On 28 Sep 2018, at 11:31, Joni Orponen wrote: > > On Fri, Sep 28, 2018 at 10:36 AM Just van Rossum wrote: > Following up to this: it seems line 1832 in py2app/build_app.py isn?t correct anymore, with Python 3 venv behavior: > > if os.path.exists(os.path.join(sys.prefix, ".Python")): > > Not sure how to fix it. If I force this code path to be taken, I get an AttributeError on sys.real_prefix, so I guess a bit more needs to be done. > > Use py2app via setup.py. That?s exactly what I do... Just From ronaldoussoren at mac.com Thu Oct 11 11:20:01 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 11 Oct 2018 17:20:01 +0200 Subject: [Pythonmac-SIG] [ANN] py2app 0.18 Message-ID: Hi, I?ve released py2app 0.18 on PyPI. The only change w.r.t. 0.17 is a recipe for ?six.moves?, that should also work when the ?six? library has been vendored by some other package. Ronald From ronaldoussoren at mac.com Tue Oct 16 15:58:28 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 16 Oct 2018 21:58:28 +0200 Subject: [Pythonmac-SIG] [ANN] PyObjC 5.1 Message-ID: <61CDDDEF-9C0C-4CD6-9484-0B54C09424AC@mac.com> Hi, I?ve pushed PyObjC 5.1 to PyPI. This is a minor feature release. The most interesting change is that the Objective-C proxies for builtin Python types now support NSSecureCoding. The full list of changes: Xcode 10 ?GM? contains one difference from the last beta: the constant MLComputeUnitsCPUAndGPU in the CoreML bindings. #222: Add a proxy for C?s ?FILE*? type on Python 3. This is not necessary on Python 2 because the default IO stack on Python 2 already uses FILE* internally. This proxy type is very minimal and shouldn?t not be used for general I/O. Bindings are up-to-date w.r.t. Xcode 10.1 (beta) Updated the support code for framework wrappers to be able to emit deprecation warnings on the first import of a deprecated constants (functions and methods will only raise a deprecation warning when called). This is just an infrastructure change, the actual framework bindings do not yet contain the information used to emit deprecation warnings. Add metadata for deprecation warnings to the ?Contacts? framework #252: Import ABCs from collections.abc instead of collections because the latter is deprecated. #180, #251: Instances of most builtin value types and sequences (int, float, str, unicode, tuple, list, set, frozenset and dict) can now be written to archives that require secureCoding. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Oct 28 01:27:43 2018 From: glyph at twistedmatrix.com (Glyph) Date: Sat, 27 Oct 2018 22:27:43 -0700 Subject: [Pythonmac-SIG] ctypes MemoryError when attempting to launch notarized app Message-ID: I adjusted my code-signing to use the new, stricter requirements imposed by app notarization. I managed to get it successfully notarized, but the app is now non-functional as a result: at startup, I get: Traceback (most recent call last): File "my.app/Contents/Resources/__boot__.py", line 93, in _setup_ctypes() File "my.app/Contents/Resources/__boot__.py", line 86, in _setup_ctypes from ctypes.macholib import dyld File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 656, in _load_unlocked File "", line 626, in _load_backward_compatible File "ctypes/__init__.pyc", line 538, in File "ctypes/__init__.pyc", line 273, in _reset_cache (If anyone wants to follow along in the traceback, this is using python.org 3.6.6.) This happens before any of my code even runs, so I can't just try to avoid ctypes. Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 , which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. Does anyone have experience with this, or ideas about what to do? -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Oct 28 05:27:09 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 28 Oct 2018 10:27:09 +0100 Subject: [Pythonmac-SIG] [Pyobjc-dev] ctypes MemoryError when attempting to launch notarized app In-Reply-To: References: Message-ID: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> > On 28 Oct 2018, at 06:27, Glyph wrote: > > I adjusted my code-signing to use the new, stricter requirements imposed by app notarization. I managed to get it successfully notarized, but the app is now non-functional as a result: at startup, I get: > > Traceback (most recent call last): > File "my.app/Contents/Resources/__boot__.py", line 93, in > _setup_ctypes() > File "my.app/Contents/Resources/__boot__.py", line 86, in _setup_ctypes > from ctypes.macholib import dyld > File "", line 971, in _find_and_load > File "", line 955, in _find_and_load_unlocked > File "", line 656, in _load_unlocked > File "", line 626, in _load_backward_compatible > File "ctypes/__init__.pyc", line 538, in > File "ctypes/__init__.pyc", line 273, in _reset_cache > > (If anyone wants to follow along in the traceback, this is using python.org 3.6.6.) On what version of macOS? I expect 10.14 because that?s the only release that actually knows about notarization, but enabling this feature might also affect how the app is signed. > > This happens before any of my code even runs, so I can't just try to avoid ctypes. You could patch the __boot__.py file before signing to see if that helps. Although this should cause problems further on, the call to _setup_ctypes should only be created when some code in your app bundle has a dependency on ctypes. > > Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. > > Does anyone have experience with this, or ideas about what to do? I?m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. With some luck there?s some entitlement or code signing option that causes this problem. What is the output of "codesign --display --verbose=4? for the application? Both with and without notarisation? Ronald > > -glyph > _______________________________________________ > Pyobjc-dev mailing list > Pyobjc-dev at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pyobjc-dev From glyph at twistedmatrix.com Sun Oct 28 14:20:47 2018 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 28 Oct 2018 11:20:47 -0700 Subject: [Pythonmac-SIG] py2app apps codesigned with "--options runtime", i.e. any notorizable app, fails to launch with ctypes error (was Re: [Pyobjc-dev] ctypes MemoryError when attempting to launch notarized app) In-Reply-To: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> Message-ID: > On Oct 28, 2018, at 2:27 AM, Ronald Oussoren wrote: > > > >> On 28 Oct 2018, at 06:27, Glyph wrote: >> >> I adjusted my code-signing to use the new, stricter requirements imposed by app notarization. I managed to get it successfully notarized, but the app is now non-functional as a result: at startup, I get: >> >> Traceback (most recent call last): >> File "my.app/Contents/Resources/__boot__.py", line 93, in >> _setup_ctypes() >> File "my.app/Contents/Resources/__boot__.py", line 86, in _setup_ctypes >> from ctypes.macholib import dyld >> File "", line 971, in _find_and_load >> File "", line 955, in _find_and_load_unlocked >> File "", line 656, in _load_unlocked >> File "", line 626, in _load_backward_compatible >> File "ctypes/__init__.pyc", line 538, in >> File "ctypes/__init__.pyc", line 273, in _reset_cache >> >> (If anyone wants to follow along in the traceback, this is using python.org 3.6.6.) > > On what version of macOS? I expect 10.14 because that?s the only release that actually knows about notarization, but enabling this feature might also affect how the app is signed. ProductName: Mac OS X ProductVersion: 10.14 BuildVersion: 18A391 >> This happens before any of my code even runs, so I can't just try to avoid ctypes. > > You could patch the __boot__.py file before signing to see if that helps. Although this should cause problems further on, the call to _setup_ctypes should only be created when some code in your app bundle has a dependency on ctypes. I'll give that a shot. >> >> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 , which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >> >> Does anyone have experience with this, or ideas about what to do? > > I?m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. As I see it, there's 2 problems here: py2app's __boot__.py should fail more gracefully if initializing ctypes doesn't work, since not everybody needs ctypes. Shall I file this on the tracker? ctypes itself should address whatever eldritch hideousness is causing this; in addition to the windows security layer stuff I found, grsecurity TPE causes the same traceback: https://bugs.python.org/issue28429 > With some luck there?s some entitlement or code signing option that causes this problem. What is the output of "codesign --display --verbose=4? for the application? Both with and without notarisation? Sorry, my original message was not clear. App notarization itself is not the problem, it's the "stricter requirements" that I ambiguously referenced. The requirement in question is the '--options runtime' flag passed to 'codesign'. So you can just codesign an app (even with an ad-hoc identity, you technically could do this without even having a valid cert, although the way one generates one of those escapes me) with the 'runtime' option, you can reproduce this. So if I sign my app like this: #!/bin/bash find "${NAME}.app" -iname '*.so' -or -iname '*.dylib' | while read libfile; do codesign --sign "${IDENTITY}" \ --deep "${libfile}" \ --force \ --options runtime; done; codesign --sign "${IDENTITY}" \ --deep "${NAME}.app" \ --force \ --options runtime; and then run it as "./${NAME}.app/Contents/MacOS/${NAME}". I immediately get the traceback given above. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Oct 28 14:47:04 2018 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 28 Oct 2018 11:47:04 -0700 Subject: [Pythonmac-SIG] py2app apps codesigned with "--options runtime", i.e. any notorizable app, fails to launch with ctypes error (was Re: [Pyobjc-dev] ctypes MemoryError when attempting to launch notarized app) In-Reply-To: References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> Message-ID: <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> > On Oct 28, 2018, at 11:20 AM, Glyph wrote: > > > >> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren > wrote: > >>> >>> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 , which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >>> >>> Does anyone have experience with this, or ideas about what to do? >> >> I?m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. > > The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. > > The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. On that note: more good news. While I haven't round-tripped through notarization again yet, this is a bit less dire than it first appeared. If I prevent the import of ctypes with an `import sys; sys.modules['ctypes'] = None`, and add a 'sed' script to my build process to prevent _setup_ctypes from running in __boot__, then the app launches again. Apparently my app doesn't actually need ctypes. The problem seems to be that Twisted includes a ctypes import; modulegraph sees this and thinks there is a hard dependency, and inserts the ctypes setup blob into __boot__. However, this is a conditional import, and it's for Windows support anyway. (There also seem to be problems with cffi-using libraries, but not other shared objects, so maybe this is a bug in libffi; however, these don't interfere with py2app itself starting up.) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Oct 28 16:48:13 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 28 Oct 2018 21:48:13 +0100 Subject: [Pythonmac-SIG] py2app apps codesigned with "--options runtime", i.e. any notorizable app, fails to launch with ctypes error (was Re: [Pyobjc-dev] ctypes MemoryError when attempting to launch notarized app) In-Reply-To: <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> Message-ID: <508550C4-F0FA-46D0-8995-A6C4948B95FB@mac.com> > On 28 Oct 2018, at 19:47, Glyph wrote: > > > >> On Oct 28, 2018, at 11:20 AM, Glyph > wrote: >> >> >> >>> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren > wrote: >> >>>> >>>> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 , which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >>>> >>>> Does anyone have experience with this, or ideas about what to do? >>> >>> I?m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. >> >> The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. >> >> The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. > > On that note: more good news. While I haven't round-tripped through notarization again yet, this is a bit less dire than it first appeared. If I prevent the import of ctypes with an `import sys; sys.modules['ctypes'] = None`, and add a 'sed' script to my build process to prevent _setup_ctypes from running in __boot__, then the app launches again. > > Apparently my app doesn't actually need ctypes. Good to hear that. > > The problem seems to be that Twisted includes a ctypes import; modulegraph sees this and thinks there is a hard dependency, and inserts the ctypes setup blob into __boot__. However, this is a conditional import, and it's for Windows support anyway. Hmm?. I wonder what?s the best way forward here. I could add on option to disable ctypes support, but that is a kludge. A weak importing hook (something like the never withdrawn PEP 369) could execute this code only when actually needed, but I have no idea how hard it would be to implement this. > > (There also seem to be problems with cffi-using libraries, but not other shared objects, so maybe this is a bug in libffi; however, these don't interfere with py2app itself starting up.) Interesting? I haven?t had complaints about PyObjC yet, and that also uses libffi. I wonder what the ?hardened runtime? option actually does and enforces. In 3.7 the line in ctypes/__init__.py that causes the exception is a call that creates a dummy C function, and likely triggers the first allocation for storing a libffi closure which could be something the hardened runtime doesn?t like (being writeable + executable memory). P.S. I just noticed that the traceback in your initial message doesn?t include the actual exception, just the traceback. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Oct 28 16:55:31 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 28 Oct 2018 21:55:31 +0100 Subject: [Pythonmac-SIG] py2app apps codesigned with "--options runtime", i.e. any notorizable app, fails to launch with ctypes error (was Re: [Pyobjc-dev] ctypes MemoryError when attempting to launch notarized app) In-Reply-To: References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> Message-ID: <56078A83-A2AB-4CFE-9DC2-2F65EE5F013E@mac.com> > On 28 Oct 2018, at 19:20, Glyph wrote: > > > >>> >>> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 , which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >>> >>> Does anyone have experience with this, or ideas about what to do? >> >> I?m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. > > The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. Yup. That?s something that worries me as well, and not just for Python apps. Not being able to run your own code without paying Apple for the privilege is not something I look forward to. I?m still hoping that the option to run unsigned code doesn?t go away. > > The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. > > As I see it, there's 2 problems here: > > py2app's __boot__.py should fail more gracefully if initializing ctypes doesn't work, since not everybody needs ctypes. Shall I file this on the tracker? Yes please. I?d prefer a solution that doesn?t involve ignoring errors, but that?s probably the easiest fix for now. What?s the exception you?re getting? Tweaking the code in py2app/bootstrap/ctypes_setup.py to ignore that exception would be trivial. > ctypes itself should address whatever eldritch hideousness is causing this; in addition to the windows security layer stuff I found, grsecurity TPE causes the same traceback: https://bugs.python.org/issue28429 >> With some luck there?s some entitlement or code signing option that causes this problem. What is the output of "codesign --display --verbose=4? for the application? Both with and without notarisation? > > Sorry, my original message was not clear. App notarization itself is not the problem, it's the "stricter requirements" that I ambiguously referenced. The requirement in question is the '--options runtime' flag passed to 'codesign'. So you can just codesign an app (even with an ad-hoc identity, you technically could do this without even having a valid cert, although the way one generates one of those escapes me) with the 'runtime' option, you can reproduce this. > > So if I sign my app like this: > > #!/bin/bash > find "${NAME}.app" -iname '*.so' -or -iname '*.dylib' | > while read libfile; do > codesign --sign "${IDENTITY}" \ > --deep "${libfile}" \ > --force \ > --options runtime; > done; > > codesign --sign "${IDENTITY}" \ > --deep "${NAME}.app" \ > --force \ > --options runtime; > > and then run it as "./${NAME}.app/Contents/MacOS/${NAME}". I immediately get the traceback given above. Great. That should make it easier for me to reproduce the issue. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Oct 28 17:57:38 2018 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 28 Oct 2018 14:57:38 -0700 Subject: [Pythonmac-SIG] py2app apps codesigned with "--options runtime", i.e. any notorizable app, fails to launch with ctypes error (was Re: [Pyobjc-dev] ctypes MemoryError when attempting to launch notarized app) In-Reply-To: <508550C4-F0FA-46D0-8995-A6C4948B95FB@mac.com> References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> <508550C4-F0FA-46D0-8995-A6C4948B95FB@mac.com> Message-ID: <3A5D116C-277B-44EF-9660-EC74601C1886@twistedmatrix.com> > On Oct 28, 2018, at 1:48 PM, Ronald Oussoren wrote: > > > >> On 28 Oct 2018, at 19:47, Glyph wrote: >> >> >> >>> On Oct 28, 2018, at 11:20 AM, Glyph wrote: >>> >>> >>> >>>> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren wrote: >>> >>>>> >>>>> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >>>>> >>>>> Does anyone have experience with this, or ideas about what to do? >>>> >>>> I?m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. >>> >>> The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. >>> >>> The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. >> >> On that note: more good news. While I haven't round-tripped through notarization again yet, this is a bit less dire than it first appeared. If I prevent the import of ctypes with an `import sys; sys.modules['ctypes'] = None`, and add a 'sed' script to my build process to prevent _setup_ctypes from running in __boot__, then the app launches again. >> >> Apparently my app doesn't actually need ctypes. > > Good to hear that. > >> >> The problem seems to be that Twisted includes a ctypes import; modulegraph sees this and thinks there is a hard dependency, and inserts the ctypes setup blob into __boot__. However, this is a conditional import, and it's for Windows support anyway. > > Hmm?. I wonder what?s the best way forward here. I could add on option to disable ctypes support, but that is a kludge. A weak importing hook (something like the never withdrawn PEP 369) could execute this code only when actually needed, but I have no idea how hard it would be to implement this. > > >> >> (There also seem to be problems with cffi-using libraries, but not other shared objects, so maybe this is a bug in libffi; however, these don't interfere with py2app itself starting up.) > > Interesting? I haven?t had complaints about PyObjC yet, and that also uses libffi. > > I wonder what the ?hardened runtime? option actually does and enforces. In 3.7 the line in ctypes/__init__.py that causes the exception is a call that creates a dummy C function, and likely triggers the first allocation for storing a libffi closure which could be something the hardened runtime doesn?t like (being writeable + executable memory). Interesting. Perhaps what I want is simply https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory then? Any chance you know how to jam that into a `codesign` command line somehow? :-) > P.S. I just noticed that the traceback in your initial message doesn?t include the actual exception, just the traceback. Oh; it?s ?MemoryError?, no exception message. > Ronald > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Oct 28 19:56:49 2018 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 28 Oct 2018 16:56:49 -0700 Subject: [Pythonmac-SIG] notarization works! (was Re: not that) In-Reply-To: <3A5D116C-277B-44EF-9660-EC74601C1886@twistedmatrix.com> References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> <508550C4-F0FA-46D0-8995-A6C4948B95FB@mac.com> <3A5D116C-277B-44EF-9660-EC74601C1886@twistedmatrix.com> Message-ID: <757B86F1-E016-4E3F-B256-E6D0399E9927@twistedmatrix.com> > On Oct 28, 2018, at 2:57 PM, Glyph wrote: > >> I wonder what the ?hardened runtime? option actually does and enforces. In 3.7 the line in ctypes/__init__.py that causes the exception is a call that creates a dummy C function, and likely triggers the first allocation for storing a libffi closure which could be something the hardened runtime doesn?t like (being writeable + executable memory). > > Interesting. Perhaps what I want is simply https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory then? Any chance you know how to jam that into a `codesign` command line somehow? :-) > Thank you so much for this tip, Ronald! This was much easier than I anticipated, and things are working now! The relevant entitlements file is literally just: com.apple.security.cs.allow-unsigned-executable-memory I dropped that in a file, added `--entitlements=$THAT_FILE.plist` to my codesign invocations, removed all my workarounds for ctypes et. al. (except for the hard-coded 'import _cffi_backend' still necessary to convince modulegraph to include enough code for SSL to work), and then tried launching my app. Success! Then I tried notarizing it: also success! Time permitting, I'll be updating my blog post at https://glyph.twistedmatrix.com/2018/01/shipping-pygame-mac-app.html with this information, and possibly publishing the now unfortunately somewhat complex tooling I use to do signing now. So I don't know if I'm the first to do this, but looking at the archives for these lists I seem to be the first to report it: you can successfully codesign and notarize apps created with py2app and python 3.6! It seems to me that whatever "MAP_JIT" is (an mmap flag, I'm guessing?) libffi needs to be using it for the memory it places synthetic closures into, so that this entitlement won't be necessary with some future version of Python. But it looks like Apple is not pushing particularly hard to deprecate this one right now, thank goodness :-). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Oct 29 01:26:56 2018 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 28 Oct 2018 22:26:56 -0700 Subject: [Pythonmac-SIG] [ANN] PyObjC 5.0 In-Reply-To: <9DF47F0A-FCDC-4601-9459-4F5D359939FF@mac.com> References: <5125878D-F7B7-40FD-AA9F-F5C0E06463A4@mac.com> <9DF47F0A-FCDC-4601-9459-4F5D359939FF@mac.com> Message-ID: <43A406B0-4683-466C-ADF7-A78CC74F61DA@twistedmatrix.com> > On Sep 21, 2018, at 4:50 AM, Ronald Oussoren wrote: > >> P.S.: speaking of build gymnastics, does anyone on this list happen to have a way to use CocoaPods or something similar to pull in open source Cocoa frameworks to a PyObjC application? > > I haven?t looked into this yet. I guess the hardest part is creating the metadata for constants and pointer arguments. Could you file and issue for this? > Sorry it took a minute to get to this :). Filed here: https://bitbucket.org/ronaldoussoren/py2app/issues/255/tooling-and-or-documentation-to-integrate -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Mon Oct 29 02:58:06 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 29 Oct 2018 07:58:06 +0100 Subject: [Pythonmac-SIG] [Pyobjc-dev] notarization works! (was Re: not that) In-Reply-To: <757B86F1-E016-4E3F-B256-E6D0399E9927@twistedmatrix.com> References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> <508550C4-F0FA-46D0-8995-A6C4948B95FB@mac.com> <3A5D116C-277B-44EF-9660-EC74601C1886@twistedmatrix.com> <757B86F1-E016-4E3F-B256-E6D0399E9927@twistedmatrix.com> Message-ID: <644B8699-9E41-4CCF-9957-7D838798A74C@mac.com> > On 29 Oct 2018, at 00:56, Glyph wrote: > > > >> On Oct 28, 2018, at 2:57 PM, Glyph > wrote: >> >>> I wonder what the ?hardened runtime? option actually does and enforces. In 3.7 the line in ctypes/__init__.py that causes the exception is a call that creates a dummy C function, and likely triggers the first allocation for storing a libffi closure which could be something the hardened runtime doesn?t like (being writeable + executable memory). >> >> Interesting. Perhaps what I want is simply https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory then? Any chance you know how to jam that into a `codesign` command line somehow? :-) >> > > Thank you so much for this tip, Ronald! This was much easier than I anticipated, and things are working now! Great. > > The relevant entitlements file is literally just: > > > "> > > > com.apple.security.cs.allow-unsigned-executable-memory > > > > > I dropped that in a file, added `--entitlements=$THAT_FILE.plist` to my codesign invocations, removed all my workarounds for ctypes et. al. (except for the hard-coded 'import _cffi_backend' still necessary to convince modulegraph to include enough code for SSL to work), and then tried launching my app. Which package needs _cffi_backend? I can add a recipe for that to py2app to do this automagically. > Success! Then I tried notarizing it: also success! Time permitting, I'll be updating my blog post at https://glyph.twistedmatrix.com/2018/01/shipping-pygame-mac-app.html with this information, and possibly publishing the now unfortunately somewhat complex tooling I use to do signing now. > > So I don't know if I'm the first to do this, but looking at the archives for these lists I seem to be the first to report it: you can successfully codesign and notarize apps created with py2app and python 3.6! > > It seems to me that whatever "MAP_JIT" is (an mmap flag, I'm guessing?) libffi needs to be using it for the memory it places synthetic closures into, so that this entitlement won't be necessary with some future version of Python. But it looks like Apple is not pushing particularly hard to deprecate this one right now, thank goodness :-). MAP_JIT is a mmap flag that?s apparently introduced in 10.14. The slides at https://developer.apple.com/videos/play/wwdc2018/702/ mention this flag and the hardened runtime. I guess we should add this flag to the code in Modules/_ctypes/malloc_closure.c CPython) and in the similar code in PyObjC. The annoying bit is that the flag is new in 10.14, and CPython installers are created on 10.9 which means those won?t include the new flag for a long time. I?ll have to check if using MAP_JIT is ok when deploying on older macOS versions, or if the code should do a runtime version check. Ronald > > -glyph > _______________________________________________ > Pyobjc-dev mailing list > Pyobjc-dev at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pyobjc-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Mon Oct 29 04:13:59 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 29 Oct 2018 09:13:59 +0100 Subject: [Pythonmac-SIG] [Pyobjc-dev] notarization works! (was Re: not that) In-Reply-To: <644B8699-9E41-4CCF-9957-7D838798A74C@mac.com> References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> <508550C4-F0FA-46D0-8995-A6C4948B95FB@mac.com> <3A5D116C-277B-44EF-9660-EC74601C1886@twistedmatrix.com> <757B86F1-E016-4E3F-B256-E6D0399E9927@twistedmatrix.com> <644B8699-9E41-4CCF-9957-7D838798A74C@mac.com> Message-ID: <2E43FCBA-9461-48C2-89A3-D32C934EB3C1@mac.com> > On 29 Oct 2018, at 07:58, Ronald Oussoren via Pyobjc-dev wrote: > > MAP_JIT is a mmap flag that?s apparently introduced in 10.14. The slides at https://developer.apple.com/videos/play/wwdc2018/702/ mention this flag and the hardened runtime. > > I guess we should add this flag to the code in Modules/_ctypes/malloc_closure.c CPython) and in the similar code in PyObjC. The annoying bit is that the flag is new in 10.14, and CPython installers are created on 10.9 which means those won?t include the new flag for a long time. > > I?ll have to check if using MAP_JIT is ok when deploying on older macOS versions, or if the code should do a runtime version check. I filed an issue with PyObjC to ensure I don?t forget to look into this: https://bitbucket.org/ronaldoussoren/pyobjc/issues/253/use-map_fixed-on-macos-1014 . I?ll look into ctypes when I have a good solution for PyObjC. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Oct 29 13:22:05 2018 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 29 Oct 2018 10:22:05 -0700 Subject: [Pythonmac-SIG] [Pyobjc-dev] notarization works! (was Re: not that) In-Reply-To: <644B8699-9E41-4CCF-9957-7D838798A74C@mac.com> References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> <508550C4-F0FA-46D0-8995-A6C4948B95FB@mac.com> <3A5D116C-277B-44EF-9660-EC74601C1886@twistedmatrix.com> <757B86F1-E016-4E3F-B256-E6D0399E9927@twistedmatrix.com> <644B8699-9E41-4CCF-9957-7D838798A74C@mac.com> Message-ID: <1540833725.1740834.1558553816.5E07E4B9@webmail.messagingengine.com> On Sun, Oct 28, 2018, at 11:58 PM, Ronald Oussoren wrote: > > >> On 29 Oct 2018, at 00:56, Glyph wrote: >> >> >> >>> On Oct 28, 2018, at 2:57 PM, Glyph wrote:>>> >>>> I wonder what the ?hardened runtime? option actually does and >>>> enforces. In 3.7 the line in ctypes/__init__.py that causes the >>>> exception is a call that creates a dummy C function, and likely >>>> triggers the first allocation for storing a libffi closure which >>>> could be something the hardened runtime doesn?t like (being >>>> writeable + executable memory).>>> >>> Interesting. Perhaps what I want is simply >>> https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory >>> then? Any chance you know how to jam that into a `codesign` command >>> line somehow? :-)>>> >> >> Thank you so much for this tip, Ronald! This was much easier than I >> anticipated, and things are working now!> > Great. > >> >> The relevant entitlements file is literally just: >> >>> >>> >> http://www.apple.com/DTDs/PropertyList-1.0.dtd">>>> >>> >>> com.apple.security.cs.allow-unsigned-executable-memory >>> >>> >>> >> >> I dropped that in a file, added `--entitlements=$THAT_FILE.plist` to >> my codesign invocations, removed all my workarounds for ctypes et. >> al. (except for the hard-coded 'import _cffi_backend' still necessary >> to convince modulegraph to include enough code for SSL to work), and >> then tried launching my app.> > Which package needs _cffi_backend? I can add a recipe for that to > py2app to do this automagically. This may sound obvious, but: cffi :-). In my case, pyOpenSSL -> cryptography -> cffi. > >> Success! Then I tried notarizing it: also success! Time permitting, >> I'll be updating my blog post at >> https://glyph.twistedmatrix.com/2018/01/shipping-pygame-mac-app.html >> with this information, and possibly publishing the now unfortunately >> somewhat complex tooling I use to do signing now.>> >> So I don't know if I'm the first to *do* this, but looking at the >> archives for these lists I seem to be the first to *report* it: you >> can successfully codesign and notarize apps created with py2app and >> python 3.6!>> >> It seems to me that whatever "MAP_JIT" is (an mmap flag, I'm >> guessing?) libffi needs to be using it for the memory it places >> synthetic closures into, so that this entitlement won't be necessary >> with some future version of Python. But it looks like Apple is not >> pushing particularly hard to deprecate this one right now, thank >> goodness :-).> > MAP_JIT is a mmap flag that?s apparently introduced in 10.14. The > slides at https://developer.apple.com/videos/play/wwdc2018/702/ > mention this flag and the hardened runtime.> > I guess we should add this flag to the code in > Modules/_ctypes/malloc_closure.c CPython) and in the similar code in > PyObjC. The annoying bit is that the flag is new in 10.14, and > CPython installers are created on 10.9 which means those won?t include > the new flag for a long time.> > I?ll have to check if using MAP_JIT is ok when deploying on older > macOS versions, or if the code should do a runtime version check. I can't shed any light on this, but I suspect the cffi folks will also have to figure this out, and may already have some sense of how this works.? I filed an issue with them here: https://bitbucket.org/cffi/cffi/issues/391/cffi-doesnt-work-inside-a-macos-app-bundle -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Oct 30 04:24:19 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 30 Oct 2018 09:24:19 +0100 Subject: [Pythonmac-SIG] [Pyobjc-dev] notarization works! (was Re: not that) In-Reply-To: <1540833725.1740834.1558553816.5E07E4B9@webmail.messagingengine.com> References: <62C46F0A-206F-4A36-8312-217E4966BF95@mac.com> <9661BCD2-FDD5-4B15-B081-BFE1B5CFCED4@twistedmatrix.com> <508550C4-F0FA-46D0-8995-A6C4948B95FB@mac.com> <3A5D116C-277B-44EF-9660-EC74601C1886@twistedmatrix.com> <757B86F1-E016-4E3F-B256-E6D0399E9927@twistedmatrix.com> <644B8699-9E41-4CCF-9957-7D838798A74C@mac.com> <1540833725.1740834.1558553816.5E07E4B9@webmail.messagingengine.com> Message-ID: > On 29 Oct 2018, at 18:22, Glyph wrote: >>> >> >> Which package needs _cffi_backend? I can add a recipe for that to py2app to do this automagically. > > This may sound obvious, but: cffi :-). In my case, pyOpenSSL -> cryptography -> cffi. I?ll look into adding a recipe for this to py2app. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Wed Oct 31 12:48:37 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 31 Oct 2018 17:48:37 +0100 Subject: [Pythonmac-SIG] [ANN] PyObjC 5.1.1 Message-ID: Hi, I?ve pushed PyObjC 5.1.1 to PyPI. This is a very minor feature update: it adds a small number of symbols introduced in the macOS 10.14.1 SDK included with Xcode 10.1. Ronald