[Import-SIG] Loading Resources From a Python Module/Package

[Donald Stufft]

> On Jan 31, 2015, at 10:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 January 2015 at 14:48, Brett Cannon <brett at python.org> wrote:
>> Basically any API dealing with paths for loaders needs to abstract away the
>> concept of files, file-like paths, etc. and rely on using the loader API on
>> pretty much everything as a simple os.path of its own. This is why I have
>> not tried to tackle the issue of the list_contents() or some such API to
>> list modules and potentially data files as it needs to not really have a
>> concrete concept of file paths (and it really should be on finders and not
>> loaders which complicates discovery, selecting the right finder, etc.). This
>> is also why APIs wanting a file path instead of taking a file-like object
>> simply cannot play well with importlib and loaders which have alternative
>> back end storage without simply being lucky that the loader they are working
>> with uses filesystem paths (or writing out to a temp file).
> 
> At the time we designed PEP 302, the principle was very strongly to
> limit the API to the bare minimum that we knew loaders would have to
> support (you have to be able to get the content of a file, because
> that's how you load a module). This was because non-filesystem modules
> were a new concept at the time, and if we'd asked what do people need,
> everyone (ourselves included) would have automatically assumed
> "everything a filesystem can do" and we'd have ended up just designing
> a virtual filesystem API and excluding a lot of possible flexibility
> (loaders for URLs, or databases, or whatever).
> 
> Now we've had experience with PEP 302, it's clear that people aren't
> using the extra flexibility much - but they *do* miss filesystem-like
> APIs. At the moment, pkg_resources fills in the gap, but that's not
> integrated with the loader system. So I think it's probably about time
> to accept that these extensions are useful and *don't* limit
> flexibility in any practical way, and add them to the loader protocol.

I don’t think we need to even limit things to file system like loaders.
If we make the “expanded” resource APIs optional then if your non file
system loader can’t support something like listing all of the files at
a sub resource then it just doesn’t implement that. It means that maybe
every type of code won’t work with every type of loader but I think that’s
a situation that isn’t able to be remedied.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA