[Python-ideas] Distribution agnostic Python project packaging

[Nick Coghlan]

On 27 October 2016 at 22:50, James Pic <jamespic at gmail.com> wrote:
> Hi all !
>
> Ive heard some people saying it was rude to post on a mailing list without
> introducing yourself so here goes something: my name is James Pic and I've
> been developing and deploying a wide variety fof Python projects Python for
> the last 8 years, I love to learn and share and writing documentation
> amongst other things such as selling liquor.
>
> The way I've been deploying Python projects so far is probably similar to
> what a lot of people do and it almost always includes building runtime
> dependencies on the production server. So, nobody is going to congratulate
> me for that for sure but I think a lot of us have been doing so.

You're right that this is a common problem, but it also isn't a
language level problem - it's a software publication and distribution
one, and for the Python community, the folks most actively involved in
driving and/or popularising improvements in that space are those
running packaging.python.org.

While there's a fair bit of overlap between the two lists, the main
home for those discussions is over on distutils-sig:
https://mail.python.org/mailman/listinfo/distutils-sig (so called due
to the standard library module that provided Python's original
project-agnostic interface for building extension modules)

> Now I'm fully aware of distribution specific packaging solutions like
> dh-virtualenv shared by Spotify but here's my mental problem: I love to
> learn and to hack. I'm always trying now distributions and I rarely run the
> one that's in production in my company and when I'm deploying personal
> projects I like funny distributions like arch, Alpine Linux,  or interesting
> paas solutions such as cloudfoundry, openshift, rancher and many others.
>
> And I'm always facing the same problem: I have to either build runtime
> dependencies on the server, either package my thing in the platform specific
> way. I feel like I've spent a really huge amount of time doing this king of
> thing. But the java people, they have jars, and they have smooth deployments
> no matter where they deploy it.

If you're not using C extensions (the closest Python equivalent to the
typical jar use case), then ``zipapp`` should have you covered:
https://docs.python.org/3/library/zipapp.html

While the zipapp module itself is relatively new, the underlying
interpreter and import system capabilities that it relies on have been
around since Python 2.6.

> So that's the idea I'm trying to share: I'd like to b able to build a file
> with my dependencies and my project in it. I'm not sure packaging only
> Python bytecode would work here because of c modules.

For extension modules, you're facing a much harder problem than doing
the same for pure Python code (where you can just use zipapp).
However, engineering folks at Twitter put together pex, which gives
you a full virtual environment to play with on the target system, and
hence can handle extension modules as well:
https://pex.readthedocs.io/en/stable/

The only runtime dependency pex places on the target system is having
a Python runtime available.

More generally, one of the major problems we have in this area at the
moment is that a lot of the relevant information is just plain hard
for people to find, so if this is an area you're interested in, then
https://github.com/pypa/python-packaging-user-guide/issues/267 is
aiming to pull together some of the currently available information
into a more readily consumable form and is mainly waiting on a draft
PR that attempts to make the existing content more discoverable.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia