[Distutils] [ANN] pkgdata

[Bob Ippolito]

On Mar 25, 2004, at 2:37 PM, Phillip J. Eby wrote:

> At 01:28 PM 3/25/04 -0500, Bob Ippolito wrote:
>
>> On Mar 25, 2004, at 11:28 AM, Phillip J. Eby wrote:
>>
>>> At 11:11 AM 3/25/04 -0500, Bob Ippolito wrote:
>>>> As per the BOF last night, here is the pkgdata module that I 
>>>> mentioned.  Currently the only project using it is pygame.  The 
>>>> "sys.datapath" mechanism that I talked about *could actually be 
>>>> implemented* on top of this system, supposing an appropriate 
>>>> adapter was written to support it!  Remember:  pkgdata is *just a 
>>>> hook*.  It doesn't do anything other give the eventual option of 
>>>> doing something smarter than 
>>>> os.path.join(os.path.basedir(__file__), data).  When you figure out 
>>>> what something smarter is, or you change your mind about what the 
>>>> smarter solution is (inside of an app bundle vs. system wide 
>>>> install, etc.), you don't have to refactor any of the code that 
>>>> uses pkgdata.
>>>
>>> Do you plan to support the PEP 302 'get_data()' facility?
>>
>> Probably not, I don't think anyone ever implemented it, and it seems 
>> more complicated than it needs to be.  If someone really does use it, 
>> and thinks it should be implemented, please speak up.
>
> Perhaps I was unclear.  What I meant was closer to, "why not expose 
> this functionality as methods on a __loader__ attribute of the module, 
> and update PEP 302 if needed?"

Because import hooks are a pain, and this is simple.  pkgdata also, 
intentionally, doesn't need to be standardized in Python.  It uses 
protocolForURI and can (and should!) actually live inside every package 
that you use it from.

> I'd love there to be a standard facility in Python for doing this, and 
> have it be supported for import-from-zip, py2exe, and all the other 
> possible routes to importing a module.  As far as I'm aware, PEP 302 
> is the only PEP that even vaguely touches on this, so it seems to make 
> sense to me to update it.

That is a bit of a different problem.  pkgdata solves the easy problem. 
  If everyone starts using it, it will at least be possible to solve the 
hard problem, where the hard problem is intractable if all code keeps 
on using the "pragmatic" os.path.join(os.path.basedir(__file__), 
'myfile') garbage.

> As you probably know, PEAK has 'config.fileNearModule()' to do this, 
> but it doesn't have any mechanisms for dealing with non-file imports.  
> I intend to try to implement the PEP 302 route once I get around to 
> needing it.  (Although, in all honesty I'm unlikely to embrace zipfile 
> imports until there's a way to use C extensions in them, at which 
> point it becomes a cool way to do simple binary installs of large 
> packages.)

The way we do C extensions in a zip file on the Mac is to put a python 
shim in place of the extension, and load the extension from somewhere 
else.  If you were to put the actual extension in the zip file, you 
would have to extract it to disk somewhere anyway because most dynamic 
linkers won't load objects that aren't files (although it seems like 
Mac OS X 10.4 will support this).

If you're particularly interested, the extension shim file template is 
this:

def __load():
     import imp, sys, os
     for p in sys.path:
         path = os.path.join(p, "%(filename)s")
         if os.path.exists(path):
             break
     else:
         assert 0, "file not found: %(filename)s"
     mod = imp.load_dynamic("%(name)s", path)

__load()
del __load

-bob