[Distutils] Setuptools download cache

[Phillip J. Eby]

At 03:20 PM 8/22/2008 -0400, David Turner wrote:
>On Wed, 2008-08-20 at 18:39 -0400, Phillip J. Eby wrote:
> > At 04:40 PM 8/20/2008 -0400, David Turner wrote:
> > >Attached is a patch which adds download caching to setuptools.
> > >
> > >At TOPP (http://topp.openplans.org/), we use a system called fassembler
> > >to build our opencore stack.  It creates approximately a dozen
> > >virtualenvs, each with their own lib/python, and then uses setuptools to
> > >install lots of libraries.  Some of these libraries are common among
> > >multiple apps, but we install multiple copies for ease of development.
> > >And every time we rebuild, we start the whole process over again.  The
> > >major slowdown in building is downloading a bunch of things which
> > >probably haven't changed since last time we downloaded them.   This
> > >patch will let us maintain a cache of all downloads, and thus do builds
> > >much faster.
> >
> > The process I'd suggest for this use case is to build the external
> > libraries using:
> >
> >     easy_install -f cache_dir -zmaxd cache_dir lib1 lib2 ...
> >
> > This command will NOT go to the web for new versions of libraries,
> > unless you also use -U.  But it will ensure that the specified
> > libraries have suitable eggs in cache_dir.
> >
> > Then, to install a given set of libraries to a virtualenv, use:
> >
> >     easy_install -f cache_dir lib1 lib2 ...
> >
> > Or, if you really insist on multiple copies of the eggs (instead of
> > just linking to them), use:
> >
> >     easy_install -af cache_dir lib1 lib2 ...
> >
> > (which will copy the .egg files even if they could be used in place).
> >
> > Unlike your caching proposal, this approach gives you finer control
> > over which libraries to update, when.  You can also update the cache
> > without changing what's installed in a given virtualenv.
>
>I'm having a very hard time getting it working, actually.  I should note
>that I'm not using easy_install directly, but through setup.py.  This is
>because I don't want to have to list all my dependencies twice, and
>setup.py passes options on to easy_install, as I understand it.

You don't have to specify dependencies twice.  Just do:

    easy_install -f cache_dir -zmaxd cache_dir path_to_checkout

It will then build an egg from the checkout and copy it and all the 
dependencies to the cache dir.



>  Here's a simple test case:
>
>1. Create a virtualenv:
>
>$ virtualenv.py /tmp/testve
>
>2. Activate
>
>$ cd /tmp/testve
>$ . bin/activate
>
>3. Check out Cabochon.
>
>(testve)$ svn co https://svn.openplans.org/svn/Cabochon/trunk cabochon
>
>4. Try to set up
>
>(testve)$ mkdir /tmp/ec2 # the cache directory

Here, you should run easy_install -f /tmp/ec2 -zmaxd /tmp/ec2 cabochon


>(testve)$ cd cabochon
>(testve)$ python setup.py develop -f /tmp/ec2 -zmaxd /tmp/ec2

Then here, run:

    python setup.py develop -af /tmp/ec2

This will then copy any dependency eggs from the cache dir to the 
virtualenv, and set up the checkout for development.

>running develop
>Checking .pth file support in /tmp/ec2
>/tmp/testve/bin/python -E -c pass
>running egg_info
>creating Cabochon.egg-info
>writing requirements to Cabochon.egg-info/requires.txt
>writing Cabochon.egg-info/PKG-INFO
>writing top-level names to Cabochon.egg-info/top_level.txt
>writing dependency_links to Cabochon.egg-info/dependency_links.txt
>writing entry points to Cabochon.egg-info/entry_points.txt
>writing manifest file 'Cabochon.egg-info/SOURCES.txt'
>writing manifest file 'Cabochon.egg-info/SOURCES.txt'
>running build_ext
>Creating /tmp/ec2/Cabochon.egg-link (link to .)
>
>Installed /tmp/testve/cabochon
>
>Because this distribution was installed --multi-version, before you can
>import modules from this package in an application, you will need to
>'import pkg_resources' and then use a 'require()' call similar to one of
>these examples, in order to select the desired version:
>
>     pkg_resources.require("Cabochon")  # latest installed version
>     pkg_resources.require("Cabochon==0.2dev-r19871")  # this exact
>version
>     pkg_resources.require("Cabochon>=0.2dev-r19871")  # this version or
>higher
>
>
>Note also that the installation directory must be on sys.path at runtime
>for
>this to work.  (e.g. by being the application's script directory, by
>being on
>PYTHONPATH, or by being added to sys.path by your code.)
>[many more lines of this as it installs all the requirements]




>5. Try to run
>
>(testve)$ paster
>bash: paster: command not found
>
>6. Hm, that's no good.  Well, what if we just manually try to see if
>stuff is installed:
>
>(testve)$ python
>Python 2.5.2 (r252:60911, Jul 31 2008, 17:31:22)
>[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
>Type "help", "copyright", "credits" or "license" for more information.
> >>> import cabochon
>Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "cabochon/__init__.py", line 8, in <module>
>     from cabochon.config.middleware import make_app
>   File "cabochon/config/middleware.py", line 2, in <module>
>     from paste import httpexceptions
>ImportError: No module named paste
>
>Nope.

This didn't work because you only did half of what I said; you have 
to do the -zmaxd step to load or update the cache, and the -af step 
to actually install your target to the virtualenv.