[Python-checkins] peps: PEP 474: Updated forge.python.org proposal
nick.coghlan
python-checkins at python.org
Thu Jan 8 04:06:12 CET 2015
https://hg.python.org/peps/rev/dadbf78cddcb
changeset: 5657:dadbf78cddcb
user: Nick Coghlan <ncoghlan at gmail.com>
date: Thu Jan 08 13:05:54 2015 +1000
summary:
PEP 474: Updated forge.python.org proposal
files:
pep-0474.txt | 242 +++++++++++++++++++++++++++++++-------
1 files changed, 197 insertions(+), 45 deletions(-)
diff --git a/pep-0474.txt b/pep-0474.txt
--- a/pep-0474.txt
+++ b/pep-0474.txt
@@ -23,18 +23,11 @@
for CPython itself (see PEP 462 in relation to that).
-PEP Deferral
-============
-
-This PEP is deferred largely because I don't currently have time to work on
-it. If anyone would like to take it over, let me know.
-
-
Proposal
========
-This PEP proposes that, once the Kallithea project has made an official
-release, that a Kallithea instance be deployed as "forge.python.org".
+This PEP proposes that an instance of the self-hosted Kallithea code
+repository management system be deployed as "forge.python.org".
Individual repositories (such as the developer guide or the PEPs repository)
may then be migrated from the existing hg.python.org infrastructure to the
@@ -42,9 +35,24 @@
will need to decide whether to retain a read-only mirror on hg.python.org,
or whether to just migrate wholesale to the new location.
-This would not be a general purpose hosting site for arbitrary Python
-projects, but a more narrowly focused site akin to the existing
-hg.python.org.
+In addition to supporting read-only mirrors on hg.python.org,
+forge.python.org will also aim to support hosting mirrors on popular
+proprietary hosting sites like GitHub and BitBucket. The aim will be to
+allow users familiar with these sites to submit and discuss pull requests
+using their preferred workflow, with forge.python.org automatically bringing
+those contributions over to the master repository.
+
+Given the availability and popularity of commercially backed "free for open
+source projects" repository hosting services, this would not be a general
+purpose hosting site for arbitrary Python projects. The initial focus will be
+specifically on CPython and other repositories currently hosted on
+hg.python.org. In the future, this could potentially be expanded to
+consolidating other PSF managed repositories that are currently externally
+hosted to gain access to a pull request based workflow, such as the
+repository for the python.org Django application. As with the initial
+migrations, any such future migrations would be considered on a case-by-case
+basis, taking into account the preferences of the primary users of each
+repository.
Rationale
@@ -64,36 +72,42 @@
The key requirements proposed for a PSF provided software forge are:
-* Must support self-hosting on PSF infrastructure without ongoing fees
-* Must support Mercurial (for consistency with existing tooling)
-* Must support simple "pull request" style workflows
-* Must support online editing for simple changes
+* MUST support simple "pull request" style workflows
+* MUST support online editing for simple changes
+* MUST be backed by an active development organisation (community or
+ commercial)
-Ideally, the chosen solution would also be a fully open source application
-written in Python.
+Additional recommended requirements that are satisfied by this proposal,
+but may be negotiable if a sufficiently compelling alternative is presented:
-The requirement for self-hosting without ongoing fees rules out the
+* SHOULD support self-hosting on PSF infrastructure without ongoing fees
+* SHOULD be a fully open source application written in Python
+* SHOULD support Mercurial (for consistency with existing tooling)
+* SHOULD support Git (to provide that option to users that prefer it)
+* SHOULD allow users of git and Mercurial clients to transparently
+ collaborate on the same repository
+* SHOULD be open to customisation to meet the needs of CPython core
+ development, including providing a potential path forward for the
+ proposed migration to a core reviewer model in PEP 462
+
+
+The preference for self-hosting without ongoing fees rules out the
free-as-in-beer providers like GitHub and BitBucket, in addition to the
various proprietary source code management offerings.
-The requirement for Mercurial support not only rules out GitHub, but also
+The preference for Mercurial support not only rules out GitHub, but also
other Git-only solutions like GitLab and Gitorious.
-The requirement for online editing support rules out the Apache
+The hard requirement for online editing support rules out the Apache
Allura/HgForge combination.
-That leaves two main contenders from a technical perspective:
+The preference for a fully open source solution rules out RhodeCode.
-* `RhodeCode <https://code.rhodecode.com/rhodecode>`__
-* `Kallithea SCM <https://kallithea-scm.org/>`__
+Of the various options considered by the author of this proposal, that
+leaves `Kallithea SCM <https://kallithea-scm.org/>`__ as the proposed
+foundation for a forge.python.org service.
-The `legal uncertainty
-<http://sfconservancy.org/blog/2014/jul/15/why-kallithea/>`__ over the
-interaction between RhodeCode's current "Business Source" licensing model and
-the various GPL components it relies on currently make it unclear whether it
-is legally permissible to deploy it.
-
-By contrast, Kallithea is a full GPLv3 application (derived from the clearly
+Kallithea is a full GPLv3 application (derived from the clearly
and unambiguously GPLv3 licensed components of RhodeCode) that is being
developed under the auspices of the `Software Freedom Conservancy
<http://sfconservancy.org/news/2014/jul/04/kallithea-joins/>`__. The
@@ -106,7 +120,7 @@
Twisted, Gevent, BuildBot and PyPy.
-Perceived Benefits
+Intended Benefits
==================
The primary benefit of deploying Kallithea as forge.python.org is that
@@ -122,24 +136,97 @@
installation.
-Technical Challenges
-====================
+Sustaining Engineering Considerations
+=====================================
+
+Even with its current workflow, CPython itself remains one of the largest
+open source projects in the world (in the
+`top 2% <https://www.openhub.net/p/python/factoids#FactoidTeamSizeVeryLarge>`
+of projects tracked on OpenHub). Unfortunately, we have been significantly
+less effective at encouraging contributions to the projects that make up
+CPython's workflow infrastructure, including ensuring that our installations
+track upstream, and that wherever feasible, our own customisations are
+contributed back to the original project.
+
+As such, a core component of this proposal is to actively engage with the
+upstream Kallithea community to lower the barriers to working with and on
+the Kallithea SCM, as well as with the PSF Infrastructure team to ensure
+the forge.python.org service integrates cleanly with the PSF's infrastructure
+automation.
+
+This approach aims to provide a number of key benefits:
+
+* allowing those of us contributing to maintenance of this service to be
+ as productive as possible in the time we have available
+* offering a compelling professional development opportunity to those
+ volunteers that choose to participate in maintenance of this service
+* making the Kallithea project itself more attractive to other potential
+ users by making it as easy as possible to adopt, deploy and manage
+* as a result of the above benefits, attracting sufficient contributors both
+ in the upstream Kallithea community, and within the CPython infrastructure
+ community, to allow the forge.python.org service to evolve effectively to
+ meet changing developer expectations
+
+Some initial steps have already been taken to address these sustaining
+engineering concerns:
+
+* Tymoteusz Jankowski has been working with Donald Stufft to work out `what
+ would be involved <https://github.com/xliiv/psf-salt/tree/kallithea>`__
+ in deploying Kallithea using the PSF's Salt based infrastructure automation.
+* Graham Dumpleton and I have been working on
+ `making it easy
+<http://www.curiousefficiency.org/posts/2014/12/kallithea-on-openshift.html>
+`__ to deploy
+ demonstration Kallithea instances to the free tier of Red Hat's open source
+ hosting service, OpenShift Online. (See the comments on that post, or the
+ `quickstart issue tracker
+ <https://github.com/ncoghlan/openshift-kallithea/issues/>`__ for links to
+ Graham's follow on work)
+
+The next major step to be undertaken is to come up with a local development
+workflow that allows contributors on Windows, Mac OS X and Linux to run
+the Kallithea tests locally, without interfering with the operation of
+their own system. The currently planned approach for this is to focus on
+Vagrant, which is a popular automated virtual machine management system
+specifically aimed at developers running local VMs for testing purposes.
+The `Vagrant based development guidelines
+<http://www.openshift.org/documentation/oo_deployment_guide_vagrant.html>`__
+for
+OpenShift Origin provide an extended example of the kind of workflow this
+approach enables. It's also worth noting that Vagrant is one of the options
+for working with a local build of the `main python.org website
+<https://github.com/python/pythondotorg#using-vagrant>`__.
+
+If these workflow proposals end up working well for Kallithea, they may also
+be worth proposing for use by the upstream projects backing other PSF and
+CPython infrastructure services, including Roundup, BuildBot, and the main
+python.org web site.
+
+
+Technical Concerns and Challenges
+=================================
+
+Introducing a new service into the CPython infrastructure presents a number
+of interesting technical concerns and challenges. This section covers several
+of the most significant ones.
User account management
-----------------------
-Ideally we'd be able to offer a single account that spans all python.org
-services, including Kallithea, Roundup/Rietveld, PyPI and the back end for
-the new python.org site, but actually implementing that would be a distinct
-infrastructure project, independent of this PEP.
+Ideally we'd like to be able to offer a single account that spans all
+python.org services, including Kallithea, Roundup/Rietveld, PyPI and the
+back end for the new python.org site, but actually implementing that would
+be a distinct infrastructure project, independent of this PEP.
-A potentially simpler near term solution would be if Kallithea's user
-account management could be integrated with the existing account management
-in Roundup, similar to what was done for Rietveld. However, if that also
-turns out to be impractical in the near term, we would merely end up with
-another identity silo to be integrated at a later date, suggesting that
-this doesn't need to be considered a blocker for an initial Kallithea
-deployment.
+For the initial rollout of forge.python.org, we will likely create yet another
+identity silo within the PSF infrastructure. A potentially superior
+alternative would be to add support for `python-social-auth
+<https://python-social-auth.readthedocs.org>`__ to Kallithea, but actually
+doing so would not be a requirement for the initial rollout of the service
+(the main technical concern there is that Kallithea is a Pylons application
+that has not yet been ported to Pyramid, so integration will require either
+adding a Pylons backend to python-social-auth, or else embarking on the
+Pyramid migration in Kallithea).
Breaking existing SSH access and links for Mercurial repositories
@@ -153,6 +240,71 @@
break).
+Integration with Roundup
+------------------------
+
+Kallithea provides configurable issue tracker integration. This will need
+to be set up appropriately to integrate with the Roundup issue tracker at
+bugs.python.org before the initial rollout of the forge.python.org service.
+
+
+Accepting pull requests on GitHub and BitBucket
+-----------------------------------------------
+
+The initial rollout of forge.python.org would support publication of read-only
+mirrors, both on hg.python.org and other services, as that is a relatively
+straightforward operation that can be implemented in a commit hook.
+
+While a highly desirable feature, accepting pull requests on external
+services, and mirroring them as submissions to the master repositories on
+forge.python.org is a more complex problem, and would likely not be included
+as part of the initial rollout of the forge.python.org service.
+
+
+Transparent Git and Mercurial interoperability
+----------------------------------------------
+
+Kallithea's native support for both Git and Mercurial offers an opportunity
+to make it relatively straightforward for developers to use the client
+of their choice to interact with repositories hosted on forge.python.org.
+
+This transparent interoperability does *not* exist yet, but running our own
+multi-VCS repository hosting service provides the opportunity to make this
+capability a reality, rather than passively waiting for a proprietary
+provider to deign to provide a feature that likely isn't in their commercial
+interest. There's a significant misalignment of incentives between open
+source communities and commercial providers in this particular area, as even
+though offering VCS client choice can significantly reduce community friction
+by eliminating the need for projects to make autocratic decisions that force
+particular tooling choices on potential contributors, top down enforcement
+of tool selection (regardless of developer preference) is currently still
+the norm in the corporate and other organisational environments that produce
+GitHub and Atlassian's paying customers.
+
+Prior to acceptance, in the absence of transparent interoperability, this PEP
+should propose specific recommendations for inclusion in the CPython
+developer's guide for using git-hg to create pull requests against
+forge.python.org hosted Mercurial repositories.
+
+
+Future Implications for CPython Core Development
+================================================
+
+The workflow requirements for the main CPython development repository are
+significantly more complex than those for the repositories being discussed
+in this PEP. These concerns are covered in more detail in PEP 462.
+
+Given Guido's recommendation to replace Reitveld with a more actively
+maintained code review system, my current plan is to rewrite that PEP to
+use Kallithea as the proposed glue layer, with enhanced Kallithea pull
+requests eventually replacing the current practice of uploading patche files
+directly to the issue tracker.
+
+I've also started working with Pierre Yves-David on a `custom Mercurial
+extension <https://bitbucket.org/ncoghlan/cpydev/src/default/cpyhg.py?at=default>`__
+that automates some aspects of the CPython core development workflow.
+
+
Copyright
=========
--
Repository URL: https://hg.python.org/peps
More information about the Python-checkins
mailing list