[Python-Dev] zipimport.c broken with implicit namespace packages

[Guido van Rossum]

On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:

>
> --
> >>>>> "Brett" == Brett Cannon <brett at python.org> writes:
>
>     > I opened
>     > https://bugs.python.org/issue25711 to specifically try to
>     > fix this issue once and for all and along the way modernize
>     > zipimport by rewriting it from scratch to be more
>     > maintainable
>
>   Every time I read about impementing a custom loader:
>
> https://docs.python.org/3/library/importlib.html
>
>   I've wondered why python does not have some sort of virtual
> filesystem layer to deal with locating modules/packages/support
> files.   Virtual file systems seem like a good way to store data on a
> wide range of storage devices.
>

Yeah, but most devices already implement a *real* filesystem, so the only
time the VFS would come in handy would be for zipfiles, where we already
have a solution.


>   A VFSLoader object would interface with importlib and deal with:
>
>   - implementing a finder and loader
>
>   - Determine the actual type of file to load (.py, .pyc, .pyo,
>     __pycache__, etc).
>
>   - Do all of it's work by calling virtual functions such as:
>      * listdir(path)
>      * read(path)
>      * stat(path)  # for things like mtime, size, etc
>      * write(path, data)  # not all VFS implement this
>

Emulating a decent filesystem API requires you to implement functionality
that would never be used by an import loader (write() is an example -- many
of the stat() fields are another example). So it would just be overkill.


>   Then things like a ziploader would just inherit from VFSLoader
> implement the straight forward methods and everything should "just
> work" :).  I see no reason why every type of loader (real filesystem,
> http, ssh, sql database, etc) would not do this as well.


All those examples except "real filesystem" are of very limited practical
value.


> Leave all
> the details such as implicit namespace packages, presence of
> __init__.py[oc] files, .pth files, etc in one single
> location and put the details on how to interact with the actual
> storage device in leaf classes which don't know or care about the high
> level details.  They would not even know they are loading python
> modules.  It is just blobs of data to them.
>

Actually the import loader API is much more suitable and less work to
implement than a VFS.


>   I may try my hand at creating a prototype of this for just the
> zipimporter and see how it goes.
>

That would nevertheless be an interesting exercise -- I hope you do it and
report back.


>     > At this point the best option might be, Mike, if you do a
>     > code review for https://bugs.python.org/issue17633, even if
>     > it is simply a LGTM. I will then personally make sure the
>     > approved patch gets checked in for Python 3.6 in case the
>     > rewrite of zipimport misses the release.
>
>   Cool.  I'll see what I can do.  I was having a bit of trouble with
> the register/login part of the bug tracker.  Which is why I came
> here.  I'll battle with it one more time and see if I can get it to
> log me in.
>

If you really can't manage to comment in the tracker (which sounds unlikely
-- many people have succeeded :-) you can post your LGTM on the specific
patch here.


>   The patch should be fairly simple.  In a nutshell it just does a:
>
>   path.replace('.', '/') + '/' in two locations.  One where it checks
> for the path being a directory entry in the zip file and the second to
> return an implicit namespace path (instead of not found) if it is a
> match.   I'll check the patch on the tracker and see if it still works
> with 3.5.1.  If not I'll attach mine.


Well, Brett would like to see your feedback on the specific patch. Does it
work for you?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160102/64ba5d83/attachment.html>