Looking for tips and gotchas for working with Python 3.5 zipapp feature

[Paul Moore]

On 16 September 2016 19:48 at 21:08, Malcolm Greene <python at bdurham.com> wrote:
> Looking for tips or edge case gotchas associated with using Python 3.5's
> new zipapp feature.

It's worth pointing out that support for executing zipped applications
has been part of Python since Python 2.6 or so. The pyz files created
by the zipapp module can be used with any version of Python back to
then (assuming the code doesn't use newer features of Python, of
course).

There aren't many "gotchas" that I know of - the main one being the
one you note below about locating data files - some modules (not many
these days) assume they are living in a normal directory and locate
data files by doing path calculations based on __file__. This doesn't
work in a zipfile.

The other main limitation (not so much a gotcha as a consequence of
how the OS works) is that you can't load C extensions (pyd or so
files) from a zipfile. If you need to do that, you'll have to bundle
the C extensions to work around that limitation, but that's pretty
advanced usage.

> For those of you wondering what this feature is, see
> the end of this post for a brief background [1].
>
> Two questions in particular:
>
> 1. Are there any issues with deploying scripts that sit in sub-
>    folders beneath the directory being zipped, eg. does zipapp only
>    support a flat folder of scripts or does it recursively zip and
>    path sub-folders?

It will work fine with a folder structure. However, the way it works
is that it runs a __main__.py script which is at the root of the
zipfile, with the zipfile inserted on sys.path. So, if you have
"import foo" in your __main__.py file, foo.py must be at the root.

If you have something like the following layout in your zipfile, you
can do "import foo.bar.baz" from your __main__.py. Basically the usual
rules for imports apply.

    __main__.py
    foo
        __init__.py
        bar
            __init__.py
            baz.py

> 2. Can additional non-Python files like config files be added to a
>    zipapp without breaking them and if so, how would your script
>    reference these embedded files (by opening up the zipapp as a zip
>    archive and navigating from there?).

They can be, but as you note, you need to reference those data files
specially, you can't find them via the __file__ attribute of a module,
and path manipulation.

The stdlib API for finding data files stored in a package is
pkgutil.get_data
(https://docs.python.org/3/library/pkgutil.html#pkgutil.get_data).
Basically, if you have a "data file" called myapp.ini" in the package
directory of the "foo" package, you can access it as

    data = pkgutil.get_data('foo', 'myapp.ini')

The content of the file is returned as a bytes object.

File layout:

    __main__.py
    foo
        __init__.py
        myapp.ini
        bar
            __init__.py
            baz.py

__main__.py

    import pkgutil
    import foo.bar.baz
    data = pkgutil.get_data('foo', 'myapp.ini')
    print(data)

The data access API is unfortunately very minimal, for example there's
no way to find what data files are present, you need to know the name
in advance. There have been proposals to add a richer API, but none
are present yet.

If you need more than this, you can as you say open your zipapp as a
zipfile (but by doing so, you commit to only working if the
application is zipped, which may or may not matter to you). Something
like this:

__main__.py

    import zipfile
    import os

    my_archive = os.path.dirname(__file__)
    zf = zipfile.ZipFile(my_archive)
    ...


> Thank you,
> Malcolm

Hope this helps :-)

> [1] The zipapp feature of Python 3.5 is pretty cool: It allows you to
>     package your Python scripts in a single executable zip file. This
>     isn't a replacement for tools like PyInstaller or Py2Exe, eg. it
>     doesn't bundle the Python interpreter in the zip file, but it's a
>     clean way to distribute multi-file scripts in environments where you
>     have control over users' Python setups.

If you do want a completely standalone application, what you can do is
to write a small C program that embeds the Python interpreter, and
just starts your zipapp. Then ship that along with the "embedded"
Python distribution (available from the python.org pages) and you have
a standalone application. I'm currently working on such a wrapper -
the prototype is at https://github.com/pfmoore/pylaunch. If I can get
it into a suitable state, I may look at adding the wrapper to the
zipapp module for Python 3.7.

Paul