From ericsnowcurrently at gmail.com Fri May 27 17:39:44 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 27 May 2016 15:39:44 -0600 Subject: [Import-SIG] direct access of import state (e.g. modules replacing themselves in sys.modules) Message-ID: Currently Python code may directly access the interpreter-global import state through the sys module. Sometimes that code only reads the global import state and sometimes it also modifies it. One notable example of the latter is where modules replace themselves in sys.modules (in order to use a custom module type). The specific issue of modules replacing themselves in sys.modules has come up before and the idea of finding a more canonical mechanism to meet the underlying needs has been at the back of my mind for a while. Regardless, that's only a part of the broader issue. Recently I've been prepping for a successor to PEP 406 (import state) and ran into this broader issue. The tricky part is that, while the encapsulation of the import system should be self-contained, external direct access/modification of global import state (sys.*) _can_ result in a problematic disconnect from the encapsulated import state. For any distinct import system that isn't currently active (exposed via builtins.__import__; state exposed via sys.*), the problem is most pronounced ( or only a problem?) when a module is being exec'ed. If the module code directly accesses import state through the sys module then that module might not be aligned with the import system that is currently exec'ing it. If it modifies the import state then it will not be aligned. Here are some options I've thought of: 1. leave things the way they are (uninstalled-but-active import systems are out of luck) 2. hack builtins.__import__ and sys for the current exec 3. module code should access the current import state through some other namespace than sys a. builtins.import_system b. importlib.current_system (or importlib.system.current) c. sys.import_system I'd favor 3c, but for backward compatibility #2 would probably be necessary in addition. It may also make sense to give special consideration to the pattern of a module replacing itself in sys.modules. Thoughts? -eric From ericsnowcurrently at gmail.com Sun May 29 18:10:28 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 29 May 2016 16:10:28 -0600 Subject: [Import-SIG] direct access of import state (e.g. modules replacing themselves in sys.modules) In-Reply-To: References: Message-ID: Looking through all the code, I don't think it's worth it (for now) to worry about using an import "system" that isn't hooked up to the import state in sys. I'll still look into a more isolated import "system" abstraction later through. :) -eric On Fri, May 27, 2016 at 3:39 PM, Eric Snow wrote: > Currently Python code may directly access the interpreter-global > import state through the sys module. Sometimes that code only reads > the global import state and sometimes it also modifies it. One > notable example of the latter is where modules replace themselves in > sys.modules (in order to use a custom module type). > > The specific issue of modules replacing themselves in sys.modules has > come up before and the idea of finding a more canonical mechanism to > meet the underlying needs has been at the back of my mind for a while. > Regardless, that's only a part of the broader issue. > > Recently I've been prepping for a successor to PEP 406 (import state) > and ran into this broader issue. The tricky part is that, while the > encapsulation of the import system should be self-contained, external > direct access/modification of global import state (sys.*) _can_ result > in a problematic disconnect from the encapsulated import state. > > For any distinct import system that isn't currently active (exposed > via builtins.__import__; state exposed via sys.*), the problem is most > pronounced ( or only a problem?) when a module is being exec'ed. If > the module code directly accesses import state through the sys module > then that module might not be aligned with the import system that is > currently exec'ing it. If it modifies the import state then it will > not be aligned. > > Here are some options I've thought of: > > 1. leave things the way they are (uninstalled-but-active import > systems are out of luck) > 2. hack builtins.__import__ and sys for the current exec > 3. module code should access the current import state through some > other namespace than sys > a. builtins.import_system > b. importlib.current_system (or importlib.system.current) > c. sys.import_system > > I'd favor 3c, but for backward compatibility #2 would probably be > necessary in addition. It may also make sense to give special > consideration to the pattern of a module replacing itself in > sys.modules. > > Thoughts? > > -eric From ncoghlan at gmail.com Mon May 30 23:16:03 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 May 2016 13:16:03 +1000 Subject: [Import-SIG] direct access of import state (e.g. modules replacing themselves in sys.modules) In-Reply-To: References: Message-ID: On 29 May 2016 3:10 pm, "Eric Snow" wrote: > > Looking through all the code, I don't think it's worth it (for now) to > worry about using an import "system" that isn't hooked up to the > import state in sys. I'll still look into a more isolated import > "system" abstraction later through. :) That's basically why PEP 406 stalled - the associated GSoC project showed that the problem was more complex than I first thought (due to separate problems with passing state to import statements in the imported code and the process global nature of extension module loading). It's also worth keeping in mind that many of the recent changes to better support advanced use cases like lazy module loading and letting Cython better emulate pure Python import semantics are already still deeply mystifying to folks with more prosaic needs like "given a path, import that file as a module": http://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: