[Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Paul Moore]

On 17 February 2015 at 18:44, Barry Warsaw <barry at python.org> wrote:
> The broader question of pex, pyzaa, etc. is an important one for Python,
> IMHO.  Having a standard single-executable distribution story will help Python
> continue to compete on platforms that work beyond the distribution models
> we've come to think about.  App models, and transactional systems such as
> Ubuntu Core, etc. are gaining mindshare and Python's traditional way of
> deploying applications doesn't really fit very well, and that may discourage
> developers from using Python.
>
> Whether Python itself wants to put a stake in the ground today for that is an
> open question, but adding support in Python for extension modules in zips
> (whether via save-to-disk or new operating system APIs) would be useful to
> explore.
>
> I happen to like pex, but what I really want is something like:
>
> $ pyvenv foo
> $ source foo/bin/activate
> $ pip install coolthing
> $ pyzip -o coolthing.pyz .
> $ deactivate
> $ ./coolthing.pyz
>
> This is outside the scope of PEP 441, but if anybody at Pycon wants to explore
> this further, I'm in.

Barry, thanks for your comments. I'll pick up on the ones directly
related to the PEP separately, but I'd like to add some thoughts on
this. (I'm not at Pycon so dumping stuff in an email is the best I can
do :-))

I'd like to see a good single-file bundled application format for
Python (on Windows, there's py2exe, which is fantastic, but often I
simply don't *want* to bundle Python and the stdlib with my code). The
executable zip format is that solution, IMO. It's been around for
ages, and has core support. There are certain things that are needed
to make a compelling story, though. And it's definitely not a well
known feature, which I hope is something PEP 441 will address, at
least a bit.

Handling of C extensions in zipfiles isn't so much an executable zip
issue as a zipimport one. It would be nice to see something like this,
but I would much rather it be an update to zipimport than be bolted on
in whichever application does it. To my mind, having the core support
the mechanisms is a much more robust approach (and avoids the
possibility of the solution not being implemented cross-platform -
look at py2exe which solves the extensions in a zipfile issue, but
only for Windows).

Also, modules still fail to be zip-safe for other reasons, such as
needing data files to be real filesystem files (for example requests'
certificate bundle). Adding more support for packages to be zip-safe
would be fantastic, and would help in a lot more places than just the
executable zip format. (Although promoting the executable zip format
would help persuade projects to *use* those features rather than
assuming a filesystem). Donald Stufft has a proposal on resource APIs
for importlib that would help with this.

As far as issues specific to the executable zip format are concerned,
the main one is tool support. The zipapp module is intended as nothing
more than the bare minimum. I think that more complex tools should
live on PyPI, not in the core, for all the same reasons that pip was
developed outside of the core - the ideal tool needs time to develop,
needs a faster release cycle than core Python, etc, etc. At the
moment, the only substantial tools I know of are pyzzer and pex.
pyzzer is pretty good, IMO, but doesn't do much beyond the basic zip
building. I tried pex, but it didn't work on Windows at the time - the
problem I found is fixed, but I haven't really had the inclination to
try again (there's only so many projects I can cope with being the
Windows early adopter for :-)) One thing I would like to see, in the
light of PEP 441, would be for pex to switch to using ".pyz" as its
file extension, because that's what pex files are. (The documentation
also refers to  the support of __main__.py as "a quirk in the Python
importer" which I find particularly irritating, as it's not a quirk,
it's a deliberate feature implemented for precisely this sort of use
case!!!)

I agree that something that grabs all of the non-stdlib dependencies
of a package and bundles them up automatically would be brilliant.
I've intended to write something like that for a while now, but
haven't had the time. But I certainly think your "what I really want"
scenario above is entirely achievable. At least to the extent that the
package code and its dependencies support running from a
non-filesystem importer...

A final thought - I wonder how easy it would be to write an "unpacking
zipimporter" that worked like zipimport, but unpacked the zipfile to a
temporary directory at the start and cleaned up on program
termination? It would give the single-file benefits of zipimport, but
would be guaranteed 100% compatible with filesystem imports, at a cost
of worse startup/teardown performance, and extra disk space. It might
make a nice fallback importer for troublesome packages.

Paul