Getting rid of virtual environments with a better dependency system

[Chris Angelico]

On Fri, Nov 13, 2020 at 9:41 AM Peter J. Holzer <hjp-python at hjp.at> wrote:
>
> On 2020-11-12 10:56:45 +1100, Chris Angelico wrote:
> > On Thu, Nov 12, 2020 at 10:44 AM Dan Stromberg <drsalists at gmail.com> wrote:
> > > I do get a .../bin/backshift though, which is a bash script that
> > > knows how to start up python on the main module.  So the user need
> > > not source something at install time or at run time.
> > >
> > > The install process is ./configure and make install, where
> > > ./configure is Not autoconf, but acts sort of like an autoconf
> > > script.
> > >
> >
> > Ah, fair enough.
> >
> > Did you know that, with a vanilla venv, you can actually just run a
> > Python interpreter from within there, without sourcing anything? Very
> > handy for automated scripts.
>
> This also works with the Python interpreter in the shebang line.
>
> So if your "make install" for "mytool" creates a venv in
> /usr/local/lib/mytool/venv and then replaces the shebang in
> /usr/local/bin/mytool with "#!/usr/local/lib/mytool/venv/python", the
> user can just invoke "mytool" without caring about the virtual
> environment.
>
> If you do that for every Python script, size may be a problem, though: I
> just checked a few of my venvs, and they are between 13 and 418 MB:
> Average 81 MB.
>

Indeed. I would divide Python scripts into a few categories:

1) System scripts. If the script came from a Debian repository, it
runs in the Debian-provided Python and uses libraries installed with
apt/dpkg.

2) Scripts that don't really much care about their environment. These
will generally run in the latest master branch build of Python
(currently 3.10), and if they need anything installed, I install it
("globally") into that instance of Python. They can't interfere with
system scripts, but can interfere with each other.

3) Scripts that get their own venv. That could be because they
conflict with something else, or something else conflicts with them,
or they need specific versions of things, or anything.

At the moment, I can find a grand total of nine virtual environments
for various scripts. Some of them are because Python 3.10 can't run
the scripts in question; others probably could run, but I want a
consistent environment for my dev/staging so it's easier to push to
production. The largest env directory is 250ish MB, and they average
94MB, so my figures are fairly comparable to yours.

ChrisA