[Python-Dev] Require loaders set __package__ and __loader__

[Guido van Rossum]

On Sat, Apr 14, 2012 at 2:15 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Sat, Apr 14, 2012 at 2:56 PM, Brett Cannon <brett at python.org> wrote:
>> An open issue in PEP 302 is whether to require __loader__ attributes on
>> modules. The claimed worry is memory consumption, but considering importlib
>> and zipimport are already doing this that seems like a red herring.
>> Requiring it, though, opens the door to people relying on its existence and
>> thus starting to do things like loading assets with
>> ``__loader__.get_data(path_to_internal_package_file)`` which allows code to
>> not care how modules are stored (e.g. zip file, sqlite database, etc.).
>>
>> What I would like to do is update the PEP to state that loaders are expected
>> to set __loader__. Now importlib will get updated to do that implicitly so
>> external code can expect it post-import, but requiring loaders to set it
>> would mean that code executed during import can rely on it as well.
>>
>> As for __package__, PEP 366 states that modules should set it but it isn't
>> referenced by PEP 302. What I want to do is add a reference and make it
>> required like __loader__. Importlib already sets it implicitly post-import,
>> but once again it would be nice to do this pre-import.
>>
>> To help facilitate both new requirements, I would update the
>> importlib.util.module_for_loader decorator to set both on a module that
>> doesn't have them before passing the module down to the decorated method.
>> That way people already using the decorator don't have to worry about
>> anything and it is one less detail to have to worry about. I would also
>> update the docs on importlib.util.set_package and importlib.util.set_loader
>> to suggest people use importlib.util.module_for_loader and only use the
>> other two decorators for backwards-compatibility.
>
> +1

Funny, I was just thinking about having a simple standard API that
will let you open files (and list directories) relative to a given
module or package regardless of how the thing is loaded. If we
guarantee that there's always a __loader__ that's a first step, though
I think we may need to do a little more to get people who currently do
things like open(os.path.join(os.path.basename(__file__),
'some_file_name') to switch. I was thinking of having a stdlib
function that you give a module/package object, a relative filename,
and optionally a mode ('b' or 't') and returns a stream -- and sibling
functions that return a string or bytes object (depending on what API
the user is using either the stream or the data can be more useful).
What would we call thos functions and where would the live?

-- 
--Guido van Rossum (python.org/~guido)