[Python-Dev] zipimport.c broken with implicit namespace packages
Brett Cannon
brett at python.org
Sat Jan 2 23:43:13 EST 2016
On Sat, 2 Jan 2016, 20:42 Brett Cannon <brett at python.org> wrote:
> I just wanted to quickly say that Guido's observation as to how a VFS is
> overkill is right. Imagine implementing a loader using sqlite and you
> quickly realize that doing a dull VFS is more
>
"dull" -> "full"
than necessary to implement what import needs to function.
>
> On Sat, 2 Jan 2016, 19:42 Guido van Rossum <guido at python.org> wrote:
>
>> On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:
>>
>>>
>>> --
>>> >>>>> "Brett" == Brett Cannon <brett at python.org> writes:
>>>
>>> > I opened
>>> > https://bugs.python.org/issue25711 to specifically try to
>>> > fix this issue once and for all and along the way modernize
>>> > zipimport by rewriting it from scratch to be more
>>> > maintainable
>>>
>>> Every time I read about impementing a custom loader:
>>>
>>> https://docs.python.org/3/library/importlib.html
>>>
>>> I've wondered why python does not have some sort of virtual
>>> filesystem layer to deal with locating modules/packages/support
>>> files. Virtual file systems seem like a good way to store data on a
>>> wide range of storage devices.
>>>
>>
>> Yeah, but most devices already implement a *real* filesystem, so the only
>> time the VFS would come in handy would be for zipfiles, where we already
>> have a solution.
>>
>>
>>> A VFSLoader object would interface with importlib and deal with:
>>>
>>> - implementing a finder and loader
>>>
>>> - Determine the actual type of file to load (.py, .pyc, .pyo,
>>> __pycache__, etc).
>>>
>>> - Do all of it's work by calling virtual functions such as:
>>> * listdir(path)
>>> * read(path)
>>> * stat(path) # for things like mtime, size, etc
>>> * write(path, data) # not all VFS implement this
>>>
>>
>> Emulating a decent filesystem API requires you to implement functionality
>> that would never be used by an import loader (write() is an example -- many
>> of the stat() fields are another example). So it would just be overkill.
>>
>>
>>> Then things like a ziploader would just inherit from VFSLoader
>>> implement the straight forward methods and everything should "just
>>> work" :). I see no reason why every type of loader (real filesystem,
>>> http, ssh, sql database, etc) would not do this as well.
>>
>>
>> All those examples except "real filesystem" are of very limited practical
>> value.
>>
>>
>>> Leave all
>>> the details such as implicit namespace packages, presence of
>>> __init__.py[oc] files, .pth files, etc in one single
>>> location and put the details on how to interact with the actual
>>> storage device in leaf classes which don't know or care about the high
>>> level details. They would not even know they are loading python
>>> modules. It is just blobs of data to them.
>>>
>>
>> Actually the import loader API is much more suitable and less work to
>> implement than a VFS.
>>
>>
>>> I may try my hand at creating a prototype of this for just the
>>> zipimporter and see how it goes.
>>>
>>
>> That would nevertheless be an interesting exercise -- I hope you do it
>> and report back.
>>
>>
>>> > At this point the best option might be, Mike, if you do a
>>> > code review for https://bugs.python.org/issue17633, even if
>>> > it is simply a LGTM. I will then personally make sure the
>>> > approved patch gets checked in for Python 3.6 in case the
>>> > rewrite of zipimport misses the release.
>>>
>>> Cool. I'll see what I can do. I was having a bit of trouble with
>>> the register/login part of the bug tracker. Which is why I came
>>> here. I'll battle with it one more time and see if I can get it to
>>> log me in.
>>>
>>
>> If you really can't manage to comment in the tracker (which sounds
>> unlikely -- many people have succeeded :-) you can post your LGTM on the
>> specific patch here.
>>
>>
>>> The patch should be fairly simple. In a nutshell it just does a:
>>>
>>> path.replace('.', '/') + '/' in two locations. One where it checks
>>> for the path being a directory entry in the zip file and the second to
>>> return an implicit namespace path (instead of not found) if it is a
>>> match. I'll check the patch on the tracker and see if it still works
>>> with 3.5.1. If not I'll attach mine.
>>
>>
>> Well, Brett would like to see your feedback on the specific patch. Does
>> it work for you?
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160103/6f65a3c3/attachment-0001.html>
More information about the Python-Dev
mailing list