[Python-Dev] Better support for consuming vendored packages

[Barry Warsaw]

On Mar 22, 2018, at 09:58, Gregory Szorc <gregory.szorc at gmail.com> wrote:
> 
> Not all consumers of Python packages wish to consume Python packages in the common `pip install <package>` + `import <package>` manner. Some Python applications may wish to vendor Python package dependencies such that known compatible versions are always available.

It’s important to understand how painful vendoring is to some downstream consumers.  Debian is a good example.  There we often have to go through a lot of hoops to unvendor packages, both for policy and for good distribution practices.  The classic example is a security vulnerability in a library.  It’s the distro’s responsibility to fix that, but in the face of vendored dependencies, you can’t just patch the system package.  Now you also have to hunt down all the vendored versions and figure out if *they’re* vulnerable, etc.  It certainly doesn’t help that you can easily have vendored libraries vendoring their own dependencies.  I think I found one application in Debian once that had like 4 or 5 versions of urllib3 inside it!

You mention dependency hell for downstream consumers like a Linux distro, but this type of integration work is exactly the job of a distro.  They have to weigh the health and security of all the applications and libraries they support, so it doesn’t bother me that it’s sometimes challenging to work out the right versions of library dependencies.  It bothers me a lot that I have to (sometimes heavily) modify packages to devendorize dependencies, especially because it’s not always clearly evident that that has happened.

That said, I completely understand the desire for application and library authors to pin their dependency versions.  We’ve had some discussions in the past (not really leading anywhere) about how to satisfy both communities.  I definitely don’t go so far as to discourage global imports, and I definitely don’t like vendoring all your dependencies.  For applications distributed outside of a distro, there are lots of options, from zip apps (e.g. pex) to frozen binaries, etc.

Developers are mostly going to use pip, and maybe a requirements.txt, so I think that use case is well covered.  Downstream consumers need to be able to easily devendorize, but I think ultimately, the need to vendorize just points to something more fundamental missing from Python’s distribution and import system.

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180322/7105e669/attachment.sig>