[Import-SIG] Loading Resources From a Python Module/Package

[Paul Moore]

On 31 January 2015 at 14:48, Brett Cannon <brett at python.org> wrote:
> Basically any API dealing with paths for loaders needs to abstract away the
> concept of files, file-like paths, etc. and rely on using the loader API on
> pretty much everything as a simple os.path of its own. This is why I have
> not tried to tackle the issue of the list_contents() or some such API to
> list modules and potentially data files as it needs to not really have a
> concrete concept of file paths (and it really should be on finders and not
> loaders which complicates discovery, selecting the right finder, etc.). This
> is also why APIs wanting a file path instead of taking a file-like object
> simply cannot play well with importlib and loaders which have alternative
> back end storage without simply being lucky that the loader they are working
> with uses filesystem paths (or writing out to a temp file).

At the time we designed PEP 302, the principle was very strongly to
limit the API to the bare minimum that we knew loaders would have to
support (you have to be able to get the content of a file, because
that's how you load a module). This was because non-filesystem modules
were a new concept at the time, and if we'd asked what do people need,
everyone (ourselves included) would have automatically assumed
"everything a filesystem can do" and we'd have ended up just designing
a virtual filesystem API and excluding a lot of possible flexibility
(loaders for URLs, or databases, or whatever).

Now we've had experience with PEP 302, it's clear that people aren't
using the extra flexibility much - but they *do* miss filesystem-like
APIs. At the moment, pkg_resources fills in the gap, but that's not
integrated with the loader system. So I think it's probably about time
to accept that these extensions are useful and *don't* limit
flexibility in any practical way, and add them to the loader protocol.

Paul