[Distutils] Deployment with setuptools

[Mars]

This has been a very good discussion.  I have been doing a bit more
thinking regarding why I intended to use a packaging tool in the first
place.

I am relying on the package versioning scheme to ensure that critical
libraries line up across families of applications.  Libraries for
internal company-wide file formats are a good example of this.

I also think that it is easier for a deployers and developers to have
an object or entity that must be equivalent across all applications
for the application to be valid.  It is easier than referencing a
Subversion tag.

Given that, what would my ideal application lifecycle look like?

I would want applications to be registered to the machine, but I want
libraries to be local to the execution environment.  Sort of like
using the 'multi-version' option, but the environment name is still in
a central database.  Then I could issue the following command to
update every applicaton on the system:

% easy_install --list-registered | xargs -i easy_install -U -

I get:
- multiple independant environments
- a central database of registered environments
- a database of packages on a per-environment basis by looking at the
physical package names in each environment
- the ability to copy elements across environments easily
- I can not share elements across environments
- I can use elements from the system-wide shared repository.

It would be nice to install packages as either zip, editable, or
development mode on a per-environment basis.  Heck, why not have the
ability to install the entire environment in development mode, with
one command?

I prefer the distutils installation scheme for --home or --prefix or
--root lined up across operating systems.  But this is contrary to
distutils' general case installation scenario.  (perhaps a new
keyword, '--environment' or '--deploy' would make sense?  Or we could
use setup.cfg)

As far as general versioning and release policies, the method outlined
py the py.lib project looks to be light-weight and flexible
(micro-releases, API backwards compatability guaranteed through
testing, etc.):
http://codespeak.net/py/current/doc/why_py.html#upcoming-release-policy-api-guarantees

This works well with the automatic upgrading of micro-releases made
possible by the '1.0>=,<2.0a' version specification.

The micro-branching techniques that Ian mentioned go nicely with the
previously mentioned release policy and testing guideline.
http://divmod.org/trac/wiki/BranchBasedDevelopment

The overall process looks like a good method to guarantee that the
trunk never gets broken (which can be a  *very* annoying problem, for
the reasons the Divmod guys outline).

It is also possible that micro-branching would solve the ChangeLog
issues that Ian spoke of.  Since each micro-branch merge is for a
specific feature it would be possible to create a nice formatted
commit message, and use something like svn-to-changelog on the trunk
to get the whole ChangeLog automatically (even if the output format
from svn2cl is ugly...):
http://ch.tudelft.nl/~arthur/svn2cl/

It would be nice to write my code in an entirely version-agnostic
manner by relying on the environment entry points to handle the
package specification.  Then my import statements are just normal
Python code.

The idea of an environment-specific package dependancy override is
very nice.  Almost a necessity, in fact.  We have to handle the
override at resource aquisition time (before site.py is generated?)

It would be nice to hook setuptools into scons or aap or even make for
the time being, so that I could handle a more customized environment
setup.  Things like configuration templates, log file stubs,
auto-generated documentation, an environment-specific directory
structure, etc. could all be there.  (or maybe I could just check the
entire environment template out of Subversion, and let easy_install
handle filling the template with Python code.)

To support the enviroment templates we would need to somehow extract
some of the package data to a standard location.  I considered using
eager_resources for this, but it is not quite eager enough for my
requirments.  I could write an extension, 'very_eager_resources', to
do this instead.

I believe that everything here is possible using setuptools,
easy_install, and pkg_resources in it's current form, with a few
custom extensions.  There is definitely a focus on using setuptools as
an environment building and maintenance tool, or at least as something
to glue more specialized tools together.

The challenge is to create a process that is light-weight enough to be
accepted for day-to-day use, instead of re-creating something like
Java's EAR and WAR specifications.

Maris