[Python-ideas] Re: Enhancing Zipapp

[Christopher Barker]

On Mon, Jan 6, 2020 at 10:50 PM Abdur-Rahmaan Janhangeer <
arj.python at gmail.com> wrote:


> - More metadata
>

good idea, and simple.


> - Integrity check with hashing
> - Protecting the meta data
>

This could be a big challenge -- and I'm not expert, so have no idea what
the issues are.


> - Bundling 3rd party packages
>

Well, as you state below, that could make it big. but it also could make it
useful -- folks want to use environments of various sorts to keep
dependencies separate, so bundling them all up in an app would be nice.

But a thought on that -- you may be able to accomplish something similar
with conda, "conda constructor", and "conda run". -- or a new tool built
from those. The idea is that the first time you ran your "app", it would
install its dependencies, and then use them in an isolated environment. But
if the multiple apps had the same dependencies, they would share them, so
you wouldn't get major bloat on the host machine.


> Are you aiming for a bundle that can run on multiple platforms? If so,
>> then it’ll have to have a way to bundle multiple compiled extensions and
>> select the right ones at runtime.
>>
>
> According to the discussion on the Python, Be Bold thread, it became
> clear that it will be a pain to generate and will have an unnecessary
> size but sure this a most stable idea
>
> Suggesting instead to include wheels. The wheels are installed. The
> interpreter looks for packages in that app-specific folder
>

but a wheel is just as big as the installed package (at least a zipped
version) -- it's essentially the package compressed into a tarball.

If this Is essentially just zipapp with the ability to bundle dependencies,
>> then you could probably just do some sys.path hackery.
>>
>
> Could you please explain more. Thanks?
>

sure -- in your zip file, you have a "dependencies" directory. the
dependencies get installed there. Then that dir gets added to sys.path at
startup. I'm not so sure o=how to do that inside a zipfile, but it could be
done *somehow*

In any case, thus seems like something you could implement, and then see if
>> people find it useful.
>>
>
> That's a nice idea to have a working demo. I'm not a security
> expert but i'll try!
>

well, you'll need a consult on the security issues -- which you would want
well reviewed anyway ;-)


> Anyone interested in this thread can view this tool
> <https://github.com/linkedin/shiv> built by LinkedIn which
> attempts dependencies bundling.
>

There you go -- you've got half the job done already :-)

But: "Unlike “conventional” zipapps, shiv packs a site-packages style
directory of your tool’s dependencies into the resulting binary, and then
at bootstrap time extracts it into a ~/.shiv cache directory."

which is how they get around the "how to add a dir in a zip file to
sys.path" -- but I'll bet someone could hack that to no be neccesary....

-CHB

-- 
Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython