[New-bugs-announce] [issue36085] Enable better DLL resolution

[Steve Dower]

New submission from Steve Dower <steve.dower at python.org>:

So the fundamental problem is that the default DLL search path on Windows changes in various contexts, and the only consistent approach is the most difficult to support with current packaging tools. The result is .pyd files that need to resolve .dll dependencies from directories *other* than where the .pyd file is located.

Here's a generic scenario:
* my_package.subpackage1.my_module is implemented as my_package/subpackage1/my_module.pyd
* my_package.subpackage2.my_module is implemented as my_package/subpackage2/my_module.pyd
* my_module.pyd in both cases depends on HelperLib.dll
* both modules must end up with the same instance of HelperLib.dll

While there are various ways for my_modules.pyd to locate HelperLib.dll, the only totally reliable way is to put HelperLib.dll alongside my_module.pyd. However, because it is needed twice, this means two copies of the DLL, which is unacceptable.

With Python 3.8, we are *nearly* dropping support for Windows 7, and I believe we can justify dropping support for Windows 7 without KB2533625 [1], which will have been released over eight years by the time 3.8 releases. This means the DLL search path enhancements are available.


Proposal #1: CPython calls SetDefaultDllDirectories() [2] on startup and exposes AddDllDirectory() [3] via the sys or os module.

This would ensure consistency in DLL search order regardless of security settings, and modules that have their own ".libs" directory have a supported API for adding it to the search path.

Past experience of forcing a consistent search path like this is that it has broken many users who expect features like %PATH% to locate DLL dependencies to work. For security reasons, this feature is already deprecated and often disabled (see [4]), so it can't be relied upon, but it makes it impossible for a single package to modify this setting or use the supported method for adding more DLL search directories.


Proposal #2: Resolve extension modules by full name

Without this proposal, the directory structure looks like:

my_package\
-subpackage1\
--__init__.py
--my_module.pyd
--HelperLib.dll
-subpackage2\
--__init__.py
--my_module.pyd
--HelperLib.dll

After this proposal, it could look like:

my_package\
-subpackage1
--__init__.py
-subpackage2\
--__init__.py
-my_package.subpackage1.my_module.pyd
-my_package.subpackage2.my_module.pyd
-HelperLib.dll

Essentially, when searching for modules, allow going up the package hierarchy and locating a fully-qualified name at any level of the import tree.

Note that since "import my_package.subpackage1.my_module" implies both "import my_package" and "import my_package.subpackage1", those have to succeed, but then the final part of the import would use subpackage1.__path__ to look for "my_module.pyd" and my_package.__path__ to look for "my_package.subpackage1.my_module.pyd".

This allows all extension modules to be co-located in the one (importable) directory, along with a single copy of any shared dependencies.

[1]: https://go.microsoft.com/fwlink/p/?linkid=217865
[2]: https://docs.microsoft.com/windows/desktop/api/libloaderapi/nf-libloaderapi-setdefaultdlldirectories
[3]: https://docs.microsoft.com/windows/desktop/api/libloaderapi/nf-libloaderapi-adddlldirectory
[4]: https://docs.microsoft.com/windows/desktop/Dlls/dynamic-link-library-search-order

----------
assignee: steve.dower
components: Windows
messages: 336349
nosy: brett.cannon, eric.snow, ncoghlan, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Enable better DLL resolution
type: enhancement
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36085>
_______________________________________