A detailed description on virtualenv's packaging dependecies? (pip, easy_install, wheel, setuptools, distlib, etc.)

[Paul Moore]

On Friday, 13 January 2017 09:27:21 UTC, haraldn... at gmail.com  wrote:
> I was working on a bugfix for Virtualenv, regarding very long shebang lines that are breaking things. In the process, I realized that if I want really fix to my particular issue it likely needs to be done on the lower level of Python package management. I started with pip, moved to setuptools and now set my sight on distlib.
> 
> Can someone describe the specific dependencies of all the *packaging* libraries that `virtualenv` uses? And the dependencies between them?
> 
> I believe that virtualenv directly imports `pip`, `easy_install` and `wheel`. Those in turn import `setuptools` and `distlib`. Am I so lucky as to assume that distlib in the lowest-level library used by all the rest in virtualenv for packages?

I believe distlib is the lowest level. But I think you're looking at the problem wrongly. From the issue you reported, the problem you're hitting is that your OS limits the length of paths you can use in a "shebang" line, and the pathname for your virtualenv exceeds that limit. And as a result, script wrappers for Python modules installed in the virtualenv fail because the shebang line is too long.

So it's not virtualenv creating the environment that needs to be fixed, but *anything* that installs wrappers that point to the environment Python with a shebang line (wherever they are installed to).

By patching distlib, you'll catch the case of installing packages into the virtualenv using pip and distil (once they update distlib). But I don't think you will catch easy_install (which uses setuptools, which uses its own wrapper script) or setup.py install (which again most likely uses setuptools). And nothing will catch 3rd party tools that do their own thing.

So if you want to try to minimise the impact of the OS limitation, you can patch both distlib and setuptools and probably get most things (actually, there's probably not much that doesn't use pip, so just patching distlib will probably get the majority of cases). You'll have to manually apply the workaround to any scripts you write yourself that you want to run with the virtualenv's Python.

Or you can just accept the limitation and use shorter pathnames.

If you're willing to accept the limitation, and it's possible to somehow determine from the OS what it allows, then getting virtualenv to warn if you use a "too long" pathname might be an alternative solution.

Paul