[Catalog-sig] Continuous deployment, in-house PyPi repo and artifact promotion

[James Carpenter]

1) Which PyPi repository servers are the most mature and feature rich, and
therefore the best choice at the moment?

2) Do any of the PyPi repository servers support managing multiple local
repositories? If yes, which ones? If no, do you have any recommendations on
how to most easily support the artifact promotion needs of continuous
deployment within the Python build ecosystem?

=================================
Background:
As part of an effort to implement continuous deployment a fellow traveler
and I are attempting to workout a good solution for an in-house PyPI
repository manager. Support of the PyPI XML-RPC API and artifact promotion
appear to be critical requirements of any solution.

We have tried using Artifactory, but unfortunately this only gives us
support for "simple" PyPI, without the XML-RPC API distutils/pip uses to
support searches.
This is disappointing, because in other respects Artifactory and Nexus
provide very complete mature support. Most importantly Artifactory and
Nexus support the ability to host multiple local repositories and proxy
external ones. By leveraging virtual repositories within Artifactory or
Nexus one can easily inter-weave results from various local repositories as
per configurable precedence rules.

The ability to have the build tooling pull artifacts from multiple internal
repositories turns out to be very important from a continuous deployment
standpoint. Effective continuous deployment typically treats every build
artifact as a potential release candidate. As a release candidate marches
through each gauntlet of tests (unit, integration, load, manual, etc.) it
is promoted to the next internal repository. This is in contrast to the
approach of using SNAPSHOTS or the like which fails the rule of giving each
code check-in the traceability necessary to be a potential
release candidate. Whether the build tool itself (distutils, Ivy, Maven,
etc.), the repository manager (Artifactory, Nexus, CheeseShop, etc.) or
something else knows how to overlay artifacts is an implementation detail,
but some tool has to do the job. Similarly the details of how a build
artifact is promoted from one local repository to another is also an
implementation detail with a variety of possible solutions.

Recognizing support for the PyPI XMLRPC API tends to be a big deal at
scale, we are looking into using CheeseShop,  Crate.io, or similar to
address our needs. Another choice may be to place a read-only solution in
front of Artifactory that adds support for the PyPI XMLRPC API. I am hoping
members of this reading list can help point us in the right direction.

Thanks for the time and effort you have spent reading this rather long
post. I look forward to your responses.

Sincerely,
James Carpenter
jcarpenter621 at yahoo dot com

P.S.: As you may have guessed I am a visitor from the land of Java where
the Sun has now set and the Seth lord reigns. (I know a Monty Python
reference would be better, but I am a bit ignorant in that regard.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20130221/664dffe1/attachment.html>