[pydotorg-www] State of the infrastructure project

[Noah Kantrowitz]

Just wanted to send out a quick update to all parties about the progress of the infrastructure migration project.

First I would like to welcome three new volunteers to the effort whom many of you likely already know; Jannis Leidel is a long-time international man of packaging mystery and has already been helping to run PyPI, Eric Holscher is a devops-in-residence at Urban Airship in Portland, and Benjamin Smith is the lead server wrangler for Pictage/ShootQ. I look forward to working with all three in the coming weeks and moths.

Next, hosting has not been fully resolved, but I think most of my questions have been addressed with OSUOSL and Cotegix. At OSUOSL we are looking at hosting two physical machines running Ganeti and linked to the OSL Ganeti management backplane (though we may choose to bypass that and use the Ganeti API directly for management). At Contegix we will just operate in their public cloud, which has a custom (but well documented) API for management. In both cases the physical machines and native OSes will be managed by the teams at their respective hosting providers. No formal plans have been made with either host at this time.

On the VM side, there is still an open question about who will be managing things. Some want to see a full-time sysadmin(s) retained by the PSF to manage and maintain the individual VMs, but I don't think there is any agreement yet on that front. For now I am running under the assumption that we will be managing these systems via Chef on a volunteer basis. Several people have already agreed to wear a pager (figuratively speaking of course, do they even make pagers anymore?) and there are talks in progress with the DSF infrastructure team to establish a shared tier one rotation once the new infrastructure is in place.

I have nearly completed my service audit of ximinez and will send out that report when it is complete so people can vet it further. Some currently open questions:

1. Prospect of moving PyPI to Postgres 9 for better replication in a (closer to) HA scenario. I will post this to the catalog-sig group for further discussion after I get done reading the existing Postgres config and re-familiarizing myself with the PyPI codebase.

2. What remote sites are pulling backups of the PyPI package data from the Rsync server on ximinez? I think at least tummy.com is from discussions with Jannis, but a full list would be great.

3. Does anyone know the reason for the Webalizer config being under /usr/local/etc and using a custom cron job instead of the debian-standard locations?

4. Is Postfix on ximinez doing anything other than outbound delivery?

5. Is lpd just a legacy artifact or is there actually a printer (or something that looks like a printer) attached?

If anyone has any questions about the current state of the infrastructure project or wants to make sure something specific gets on the docket to be looked at, please don't hesitate to contact me. Thanks again for everyones' work so far towards a better future for the PSF servers!

--Noah