[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Tobias Bergkvist]

Tobias Bergkvist <tobias at bergkv.ist> added the comment:

Okay, I decided to look into how I could do dynamic loading as you suggested.

Here is a POC (executable) for using _dyld_shared_cache_contains_path when available:

```
#include <stdio.h>
#include <dlfcn.h>

void* libsystemb_handle;
typedef bool (*_dyld_shared_cache_contains_path_f)(const char* path);
_dyld_shared_cache_contains_path_f _dyld_shared_cache_contains_path;

bool _dyld_shared_cache_contains_path_fallback(const char* name) {
    return false;
}

__attribute__((constructor)) void load_libsystemb(void) {
    if (
        (libsystemb_handle = dlopen("/usr/lib/libSystem.B.dylib", RTLD_LAZY)) == NULL ||
        (_dyld_shared_cache_contains_path = dlsym(libsystemb_handle, "_dyld_shared_cache_contains_path")) == NULL
    ) {
        _dyld_shared_cache_contains_path = _dyld_shared_cache_contains_path_fallback;
    }
}

__attribute__((destructor)) void unload_libsystemb(void) {
    if (libsystemb_handle != NULL) {
        dlclose(libsystemb_handle);
    }
}

int main(int argc, char ** argv) {
    printf("Library exists in cache: %d\n", _dyld_shared_cache_contains_path(argv[1]));
}
```

A fallback function is used when _dyld_shared_cache_contains_path cannot be loaded, which always returns false. If there is no cache - the (nonexistent) cache also does not contain whatever path you pass it.

The constructor function is called when the Python extension is loaded - ensuring that _dyld_shared_cache_contains_path is defined and callable. I've read that C extension modules cannot be autoreloaded (https://ipython.org/ipython-doc/3/config/extensions/autoreload.html) - so this might mean there is no need for a deconstructor? Instead the OS would handle cleanup once the process exits?

This could be compiled on either MacOS Catalina or Big Sur, and run without problems on the other MacOS version.

Regarding the "explicit weak linking" when building on MacOS Big Sur and later; wouldn't this mean that a Big Sur build wouldn't work on Catalina?

Would you be positive towards a PR with the approach I demonstrated here?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue44689>
_______________________________________