[Python-Dev] Require loaders set __package__ and __loader__

[Glyph]

On Apr 14, 2012, at 7:59 PM, Guido van Rossum wrote:

> On Sat, Apr 14, 2012 at 5:06 PM, Christian Heimes <lists at cheimes.de> wrote:
>> Am 15.04.2012 00:56, schrieb Guido van Rossum:
>>> Well, if it's a real file, and you need a stream, that's efficient,
>>> and if you need the data, you can read it. But if it comes from a
>>> loader, and you need a stream, you'd have to wrap it in a StringIO
>>> instance. So having two APIs, one to get a stream, and one to get the
>>> data, allows the implementation to be more optimal -- it would be bad
>>> to wrap a StringIO instance around data only so you can read the data
>>> from the stream again...
>> 
>> We need a third way to access a file. The two methods get_data() and
>> get_stream() aren't sufficient for libraries that need a read file that
>> lives on the file system. In order to have real files the loader (or
>> some other abstraction layer) needs to create a temporary directory for
>> the current process and clean it up when the process ends. The file is
>> saved to the temporary directory the first time it's accessed.
> 
> Hm... Can you give an example of a library that needs a real file?
> That sounds like a poorly designed API.

Lots of C libraries use filenames or FILE*s where they _should_ be using some much more abstract things; i.e., constellations of function pointers that are isomorphic to Python's "file-like objects".  Are these APIs poorly designed?  Sure, but they also exist ;).

> Perhaps you're talking about APIs that take a filename instead of a
> stream? Maybe for those it would be best to start getting serious
> about a virtual filesystem... (Sorry, probably python-ideas stuff).

twisted.python.filepath... ;-)

>> The get_file() feature has a neat benefit. Since it transparently
>> extracts files from the loader, users can ship binary extensions and
>> shared libraries (dlls) in a ZIP file and use them without too much hassle.
> 
> Yeah, DLLs are about the only example I can think of where even a
> virtual filesystem doesn't help...

In a previous life, I was frequently exposed to proprietary game-engine things that could only load resources (3D models, audio files, textures) from actual real files, and I had to do lots of unpacking stuff either from things tacked on to a .exe or inside a zip file.  (I don't know how common this is any more in that world but I suspect "very".)

Unfortunately all the examples I can think of off the top of my head were in proprietary, now defunct code; but this is exactly the sort of polish that open-sourcing tends to apply, so I would guess problematic code in this regard would more often be invisible.

-glyph